Subword tokenization

Overview

A model that reads text through a fixed vocabulary cannot represent a word it never saw in training. The usual fallbacks, a dictionary look-up or copying the word across, break down when languages build words differently 1508.07909 §1. Subword tokenization avoids this by writing a word as a sequence of pieces the model already has, so any word can be segmented 1508.07909 §3.2.

The pieces are learned from a training corpus rather than written by hand. The variants differ in how they choose those pieces and in how much of the vocabulary they fix in advance. For the subword variants, the shared rule is that the same segmentation is applied at training and test time, so the network sees only units from its vocabulary. The exception is characters never seen in training, which can fall outside it 1508.07909 §3.2. Later models apply the same idea to one language: a Spanish encoder with one shared subword vocabulary, fixed before pre-training 2308.02976 §3; a Dutch encoder with a 30K WordPiece vocabulary 1912.09582 §2.1; an Arabic pair of encoders with a 100K vocabulary 2101.01785 §3.1; and a science model whose tokenizer splits digits and sequence notation more finely 2211.09085 §3.1.

A different route drops the fixed pieces. One multilingual toxicity model reads UTF-8 bytes and learns how to group them, so no vocabulary is built or shipped for each language 2202.11176 §1 2202.11176 §3.1.

Variants

Character n-grams

Split each word into character n-grams that do not cross word boundaries. A shortlist can keep the most frequent words whole. The paper's strongest English-German ensemble uses character bigrams with a 50,000-word shortlist (C2-50k) 1508.07909 §3 1508.07909 §4.1 1508.07909 tab. 2. Card: Neural Machine Translation of Rare Words with Subword Units.

Byte pair encoding (BPE)

Start from characters, repeatedly merge the most frequent adjacent pair into a new symbol, and apply the learned merges in their learned order to new text 1508.07909 §3.2 1508.07909 alg. 1. No shortlist is needed, and the merge count is the only hyperparameter 1508.07909 §3.2. The paper uses 59,500 merges for separate BPE and 89,500 for joint BPE on German training data 1508.07909 §4.1. Card: the same paper, linked above.

Joint BPE

Learn one set of merges over the source and target training text together, so a name is split the same way on both sides 1508.07909 §3.2. The authors tie this to fewer transliteration errors on English-Russian 1508.07909 §5.2 1508.07909 tab. 5. For Russian, they first transliterate the vocabulary into Latin letters to learn the joint merges, then map the merges back to Cyrillic 1508.07909 §3.2. Card: the same paper, linked above.

BPE through SentencePiece, one language, fixed vocabulary

Train a single-language encoder with one subword vocabulary of 31K subwords plus 1K placeholder tokens for later use, 32K in total. The same vocabulary serves the cased and the uncased model 2308.02976 §3. The vocabulary is built with SentencePiece, and the training text is Spanish only, about 3 billion words 2308.02976 §3. The card records no SentencePiece settings beyond the vocabulary size. Card: Spanish Pre-trained BERT Model and Evaluation Data.

WordPiece, 30K vocabulary, Dutch encoder

BERTje is a Dutch BERT-base encoder with a 30K WordPiece vocabulary. The paper says the vocabulary came from a SentencePiece model trained on the raw pretraining data, converted to WordPiece format 1912.09582 §2.1. The released pretraining script does something different: it trains a unigram SentencePiece model, not a WordPiece one (see Contradictions and the card). The paper's pretraining corpus is 12 GB, about 2.4B tokens, from five Dutch sources 1912.09582 §2.1. Card: BERTje: A Dutch BERT Model.

WordPiece, 100K vocabulary, Arabic encoders

ARBERT and MARBERT use the WordPiece tokenizer of BERT with a 100K-entry vocabulary 2101.01785 §3.1 2101.01785 §3.2. The paper does not say how the vocabulary was built 2101.01785 §3.1. Both models also pre-process the text first: they remove Arabic diacritics and replace every URL, user mention and hashtag with a literal token 2101.01785 §3.1 2101.01785 §3.2. Card: ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.

Byte-level input with a learned tokenizer

Read the text as UTF-8 bytes, embed each byte, and let a learned module group neighbouring bytes into latent subwords. That module, the GBST block from Charformer, considers blocks of one to four bytes at each position, scores each candidate, takes a softmax over block sizes, and mean-pools the result by a factor of two 2202.11176 §3.1 2202.11176 §4.2. Because the input is bytes, there is no vocabulary to build or ship per language, and misspellings, emoji and deliberate character substitutions reach the model unchanged 2202.11176 §1. The encoder-decoder is pre-trained to rebuild masked spans, covering 15% of bytes with spans of 20 bytes on average 2202.11176 §3.4. Two cautions. The pretraining mix is half a proprietary corpus, so the exact training data cannot be rebuilt 2202.11176 §3.4. The authors report a fine-tuning throughput of 32.9 steps per second for this model against 18.3 for a byte-level T5 Base, and say that the measure tracks serving latency. It is not a serving measurement 2202.11176 tab. 8 2202.11176 §8. Card: A New Generation of Perspective API: Efficient Multilingual Character-level Transformers.

BPE, 50K vocabulary, scientific text

Galactica uses byte-pair encoding with a 50k vocabulary, built from a random 2% of the training data 2211.09085 §4.1. Digits are split into single tokens, ASCII operators and parentheses into single characters, and the wrapped sequence types (SMILES, amino-acid and DNA sequences) character by character 2211.09085 §3.1. The paper's reason is that scientific terms and mathematics tokenise poorly under a general corpus 2211.09085 §2. The card records no ablation of these rules. Card: Galactica: A Large Language Model for Science.

When to use which

Corpus statistics below are for German training data, counts as printed 1508.07909 tab. 1.

Gaps of a few tenths of a BLEU point are within the dev-set variation the authors report 1508.07909 §4.2. Compare the variants on your own data before you choose on these numbers alone.

Implementation sketch

Our pseudocode, following the paper's algorithm. The end-of-word marker is shown as </w>, as in the listing 1508.07909 alg. 1.

# Learn merges from word frequencies  [arxiv:1508.07909 §3.2, alg. 1]
def learn_bpe(word_counts, num_merges):
    # Start from characters, with an end-of-word marker per word
    vocab = {tuple(word) + ("</w>",): n for word, n in word_counts.items()}
    merges = []
    for _ in range(num_merges):
        # Count adjacent pairs, weighted by word frequency, within words only
        pairs = count_adjacent_pairs(vocab)
        best = most_frequent(pairs)
        # Replace that pair everywhere by one new symbol  [alg. 1]
        vocab = merge_pair(vocab, best)
        merges.append(best)
    return merges

# Segment one word with the learned merges, in learned order  [arxiv:1508.07909 §3.2]
def segment(word, merges):
    symbols = tuple(word) + ("</w>",)
    rank = {pair: i for i, pair in enumerate(merges)}
    while True:
        # Adjacent pairs that were learned as merges
        applicable = [p for p in adjacent_pairs(symbols) if p in rank]
        if not applicable:
            break
        # Apply the merge that was learned earliest
        symbols = merge_pair(symbols, min(applicable, key=rank.get))
    # Drop the end-of-word marker before output  [arxiv:1508.07909 §3.2]
    return strip_end_marker(symbols)

Variants differ in where the merges are learned, per language or jointly. The BPE systems also keep infrequent symbols out of the network vocabulary, because rare symbols such as foreign-alphabet characters add noise 1508.07909 §4.1. The paper's merge counts are author-stated and tuned for its setup. Treat them as starting points. The SentencePiece variant runs the same merge idea through a different tool, and its settings are not given in the card 2308.02976 §3, so the sketch above is not a reproduction of it. The WordPiece, Galactica and byte-level variants are not reproduced by this sketch either: the cards give their vocabulary sizes and splitting rules, not their learning procedures. The byte-level learnable tokenizer learns block weights rather than merges, so its steps are in the card's pseudocode (A New Generation of Perspective API).

Contradictions

Within the first paper, the best variant depends on the metric and the language pair. Across the other papers there is no direct disagreement on numbers: they cover different languages, corpora and metrics, and none of the cards isolates its tokenizer against another's. Vocabulary sizes (30K for Dutch, 32K for Spanish, 100K for Arabic, 50K for science, 60K to 90K for NMT) are therefore not comparable, and the byte-level model has no vocabulary to compare. Naming differs: the Spanish paper calls its method BPE through SentencePiece in §3, but says WordPiece tokenization with the learned vocabulary in §5.1 2308.02976 §3 2308.02976 §5.1. The Arabic paper calls its tokenizer WordPiece 2101.01785 §3.1. The text does not settle which segmentation was used at fine-tuning time in the Spanish paper.

• English-German. Character bigrams give the best ensemble BLEU 1508.07909 tab. 2. Joint BPE gives the best rare-word and OOV unigram F1 1508.07909 tab. 2.
• English-Russian. Character bigrams and joint BPE tie on ensemble BLEU at 24.1. Character bigrams score higher as single models, 20.9 against 20.4 1508.07909 tab. 3. Joint BPE scores higher on rare-word unigram F1, 29.7 against 27.8 1508.07909 tab. 3.
• Headline gains. The 1.1 BLEU gain for English-German comes from character bigrams, not BPE. The 1.3 BLEU gain for English-Russian is matched by character bigrams as well as joint BPE, both at 24.1 ensemble BLEU. Read each gain against the system that produced it 1508.07909 tab. 2 1508.07909 tab. 3.
• Dutch vocabulary, paper against code. The paper builds a SentencePiece model and converts it to WordPiece 1912.09582 §2.1. The released script trains a unigram SentencePiece model with 30,001 pieces. The card also notes that the Hugging Face vocabulary changed in 2021, so the paper-era vocabulary is one specific revision. Both sides are stated; the card does not settle which one the released weights used.
• Dutch masking, paper against code. The paper masks all wordpieces of a word together 1912.09582 §2.2. The released code turns whole-word masking off, so each wordpiece is a separate candidate. Both sides are stated; check which one you mean before you train.

Papers

• Neural Machine Translation of Rare Words with Subword Units: introduces BPE for learned subword segmentation in NMT, and compares it with character bigrams and word-level baselines with dictionary back-off.
• BERTje: A Dutch BERT Model: uses a 30K WordPiece vocabulary, derived from SentencePiece, for a Dutch-only BERT-base encoder.
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic: uses a 100K WordPiece vocabulary for two BERT-Base encoders, one trained on formal Arabic and one on tweets.
• A New Generation of Perspective API: Efficient Multilingual Character-level Transformers: uses UTF-8 bytes and a learned Charformer-style tokenizer, with no fixed vocabulary, for a multilingual toxicity encoder.
• Galactica: A Large Language Model for Science: uses a 50K BPE vocabulary, with character-level splitting of digits, operators and sequence blocks, for a scientific language model.
• Spanish Pre-trained BERT Model and Evaluation Data: uses a 32K BPE subword vocabulary, built with SentencePiece, for a Spanish-only BERT-style encoder.