--- kind: concept slug: subword-tokenization title: Subword tokenization summary: >- Split words into learned pieces smaller than words, so a fixed-vocabulary model can still read rare and unseen words. Covers BPE and joint BPE, character bigrams, and vocabularies for Spanish, Dutch, Arabic and science, plus byte-level input with a learned tokenizer. Gains are author-reported. tags: [nlp, tokenization, subword, machine-translation, open-vocabulary] papers: ["1508.07909", "1912.09582", "2101.01785", "2202.11176", "2211.09085", "2308.02976"] related: [neural-machine-translation, language-model-pretraining, masked-language-modeling, multilingual-evaluation, text-embedding, toxicity-detection] status: draft updated_at: 2026-11-01 --- ## Overview A model that reads text through a fixed vocabulary cannot represent a word it never saw in training. The usual fallbacks, a dictionary look-up or copying the word across, break down when languages build words differently [arxiv:1508.07909 §1]. Subword tokenization avoids this by writing a word as a sequence of pieces the model already has, so any word can be segmented [arxiv:1508.07909 §3.2]. The pieces are learned from a training corpus rather than written by hand. The variants differ in how they choose those pieces and in how much of the vocabulary they fix in advance. For the subword variants, the shared rule is that the same segmentation is applied at training and test time, so the network sees only units from its vocabulary. The exception is characters never seen in training, which can fall outside it [arxiv:1508.07909 §3.2]. Later models apply the same idea to one language: a Spanish encoder with one shared subword vocabulary, fixed before pre-training [arxiv:2308.02976 §3]; a Dutch encoder with a 30K WordPiece vocabulary [arxiv:1912.09582 §2.1]; an Arabic pair of encoders with a 100K vocabulary [arxiv:2101.01785 §3.1]; and a science model whose tokenizer splits digits and sequence notation more finely [arxiv:2211.09085 §3.1]. A different route drops the fixed pieces. One multilingual toxicity model reads UTF-8 bytes and learns how to group them, so no vocabulary is built or shipped for each language [arxiv:2202.11176 §1] [arxiv:2202.11176 §3.1]. ## Variants ### Character n-grams Split each word into character n-grams that do not cross word boundaries. A shortlist can keep the most frequent words whole. The paper's strongest English-German ensemble uses character bigrams with a 50,000-word shortlist (C2-50k) [arxiv:1508.07909 §3] [arxiv:1508.07909 §4.1] [arxiv:1508.07909 tab. 2]. Card: [Neural Machine Translation of Rare Words with Subword Units](/papers/1508.07909). ### Byte pair encoding (BPE) Start from characters, repeatedly merge the most frequent adjacent pair into a new symbol, and apply the learned merges in their learned order to new text [arxiv:1508.07909 §3.2] [arxiv:1508.07909 alg. 1]. No shortlist is needed, and the merge count is the only hyperparameter [arxiv:1508.07909 §3.2]. The paper uses 59,500 merges for separate BPE and 89,500 for joint BPE on German training data [arxiv:1508.07909 §4.1]. Card: the same paper, linked above. ### Joint BPE Learn one set of merges over the source and target training text together, so a name is split the same way on both sides [arxiv:1508.07909 §3.2]. The authors tie this to fewer transliteration errors on English-Russian [arxiv:1508.07909 §5.2] [arxiv:1508.07909 tab. 5]. For Russian, they first transliterate the vocabulary into Latin letters to learn the joint merges, then map the merges back to Cyrillic [arxiv:1508.07909 §3.2]. Card: the same paper, linked above. ### BPE through SentencePiece, one language, fixed vocabulary Train a single-language encoder with one subword vocabulary of 31K subwords plus 1K placeholder tokens for later use, 32K in total. The same vocabulary serves the cased and the uncased model [arxiv:2308.02976 §3]. The vocabulary is built with SentencePiece, and the training text is Spanish only, about 3 billion words [arxiv:2308.02976 §3]. The card records no SentencePiece settings beyond the vocabulary size. Card: [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976). ### WordPiece, 30K vocabulary, Dutch encoder BERTje is a Dutch BERT-base encoder with a 30K WordPiece vocabulary. The paper says the vocabulary came from a SentencePiece model trained on the raw pretraining data, converted to WordPiece format [arxiv:1912.09582 §2.1]. The released pretraining script does something different: it trains a unigram SentencePiece model, not a WordPiece one (see Contradictions and the card). The paper's pretraining corpus is 12 GB, about 2.4B tokens, from five Dutch sources [arxiv:1912.09582 §2.1]. Card: [BERTje: A Dutch BERT Model](/papers/1912.09582). ### WordPiece, 100K vocabulary, Arabic encoders ARBERT and MARBERT use the WordPiece tokenizer of BERT with a 100K-entry vocabulary [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. The paper does not say how the vocabulary was built [arxiv:2101.01785 §3.1]. Both models also pre-process the text first: they remove Arabic diacritics and replace every URL, user mention and hashtag with a literal token [arxiv:2101.01785 §3.1] [arxiv:2101.01785 §3.2]. Card: [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785). ### Byte-level input with a learned tokenizer Read the text as UTF-8 bytes, embed each byte, and let a learned module group neighbouring bytes into latent subwords. That module, the GBST block from Charformer, considers blocks of one to four bytes at each position, scores each candidate, takes a softmax over block sizes, and mean-pools the result by a factor of two [arxiv:2202.11176 §3.1] [arxiv:2202.11176 §4.2]. Because the input is bytes, there is no vocabulary to build or ship per language, and misspellings, emoji and deliberate character substitutions reach the model unchanged [arxiv:2202.11176 §1]. The encoder-decoder is pre-trained to rebuild masked spans, covering 15% of bytes with spans of 20 bytes on average [arxiv:2202.11176 §3.4]. Two cautions. The pretraining mix is half a proprietary corpus, so the exact training data cannot be rebuilt [arxiv:2202.11176 §3.4]. The authors report a fine-tuning throughput of 32.9 steps per second for this model against 18.3 for a byte-level T5 Base, and say that the measure tracks serving latency. It is not a serving measurement [arxiv:2202.11176 tab. 8] [arxiv:2202.11176 §8]. Card: [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](/papers/2202.11176). ### BPE, 50K vocabulary, scientific text Galactica uses byte-pair encoding with a 50k vocabulary, built from a random 2% of the training data [arxiv:2211.09085 §4.1]. Digits are split into single tokens, ASCII operators and parentheses into single characters, and the wrapped sequence types (SMILES, amino-acid and DNA sequences) character by character [arxiv:2211.09085 §3.1]. The paper's reason is that scientific terms and mathematics tokenise poorly under a general corpus [arxiv:2211.09085 §2]. The card records no ablation of these rules. Card: [Galactica: A Large Language Model for Science](/papers/2211.09085). ## When to use which Corpus statistics below are for German training data, counts as printed [arxiv:1508.07909 tab. 1]. | Situation | Use | Why | |---|---|---| | Sequence length and speed matter most | BPE | BPE gives about 112 m tokens against 100 m for whole words. Character bigrams give 306 m and characters 550 m [arxiv:1508.07909 tab. 1]. The authors note that run time grows at least linearly with sequence length [arxiv:1508.07909 §4.1]. | | You want one hyperparameter and no shortlist | BPE | Only the merge count is chosen [arxiv:1508.07909 §3.2]. | | Source and target share names or differ in alphabet | Joint BPE | Keeps segmentation consistent across the two languages [arxiv:1508.07909 §3.2]. Separate BPE produced transliteration errors on English-Russian [arxiv:1508.07909 §5.2]. | | Highest English-German ensemble BLEU is the goal | Character bigrams (C2-50k) | Ensemble BLEU 25.3, against 24.7 for joint BPE [arxiv:1508.07909 tab. 2]. Single models tie at 22.8 [arxiv:1508.07909 tab. 2]. | | Rare-word and OOV unigram F1 is the goal | Joint BPE | Joint BPE has the highest rare-word and OOV unigram F1 on English-German [arxiv:1508.07909 tab. 2]. | | Pre-training a monolingual encoder that must cover cased and uncased text with one vocabulary | BPE through SentencePiece, 32K shared | One vocabulary for both casings, with reserved placeholder tokens [arxiv:2308.02976 §3]. The card records no tokenizer ablation, so the paper's results cannot be credited to the vocabulary choice. | | Pre-training a Dutch-only encoder with its own 30K vocabulary | WordPiece, 30K, as BERTje | The paper gives a SentencePiece-derived 30K vocabulary [arxiv:1912.09582 §2.1]. Its tables compare it with multilingual BERT [arxiv:1912.09582 tab. 1] [arxiv:1912.09582 tab. 2], so the gains cannot be credited to the vocabulary alone. | | Pre-training an encoder for one language, with its own vocabulary rather than a shared multilingual one | WordPiece, 100K, as ARBERT and MARBERT | The paper gives the Arabic share of mBERT's vocabulary as 5K of 110K and of XLM-R's as 14K of 250K [arxiv:2101.01785 §3.1]. It does not give the Arabic share of its own 100K vocabulary, so the results cannot be credited to the vocabulary alone. | | Input is noisy (misspellings, emoji, deliberate character substitutions) or mixes scripts, and a vocabulary per language is not wanted | Byte-level input with a learned tokenizer, as UTC | On production multilingual data, the 102M model averages 0.910 AUC-ROC, against 0.906 for mT5-base and 0.840 for a custom mBERT [arxiv:2202.11176 tab. 1]. Zero-shot on obfuscated English, it beats each baseline at every rate above 0% [arxiv:2202.11176 §6.2]. On one code-switching set, mT5-base scores higher, 0.9393 against 0.9191 [arxiv:2202.11176 tab. 3]. Test on your own mix. | | Text mixes equations, numbers and molecule, protein or DNA strings | Character-level splitting of digits, operators and sequence blocks, as in Galactica | The paper motivates this with poor tokenisation of mathematics and scientific terms [arxiv:2211.09085 §2] [arxiv:2211.09085 §3.1]. The card records no ablation, so test the rules on your own text. | | Vocabulary must be learned from a large corpus, and you do not want all of it | Learn the vocabulary from a sample | Galactica learned its 50K vocabulary from a random 2% of the training data [arxiv:2211.09085 §4.1]. The card records no comparison with a vocabulary learned from all of it. | Gaps of a few tenths of a BLEU point are within the dev-set variation the authors report [arxiv:1508.07909 §4.2]. Compare the variants on your own data before you choose on these numbers alone. ## Implementation sketch Our pseudocode, following the paper's algorithm. The end-of-word marker is shown as ``, as in the listing [arxiv:1508.07909 alg. 1]. ```pseudocode # Learn merges from word frequencies [arxiv:1508.07909 §3.2, alg. 1] def learn_bpe(word_counts, num_merges): # Start from characters, with an end-of-word marker per word vocab = {tuple(word) + ("",): n for word, n in word_counts.items()} merges = [] for _ in range(num_merges): # Count adjacent pairs, weighted by word frequency, within words only pairs = count_adjacent_pairs(vocab) best = most_frequent(pairs) # Replace that pair everywhere by one new symbol [alg. 1] vocab = merge_pair(vocab, best) merges.append(best) return merges # Segment one word with the learned merges, in learned order [arxiv:1508.07909 §3.2] def segment(word, merges): symbols = tuple(word) + ("",) rank = {pair: i for i, pair in enumerate(merges)} while True: # Adjacent pairs that were learned as merges applicable = [p for p in adjacent_pairs(symbols) if p in rank] if not applicable: break # Apply the merge that was learned earliest symbols = merge_pair(symbols, min(applicable, key=rank.get)) # Drop the end-of-word marker before output [arxiv:1508.07909 §3.2] return strip_end_marker(symbols) ``` Variants differ in where the merges are learned, per language or jointly. The BPE systems also keep infrequent symbols out of the network vocabulary, because rare symbols such as foreign-alphabet characters add noise [arxiv:1508.07909 §4.1]. The paper's merge counts are author-stated and tuned for its setup. Treat them as starting points. The SentencePiece variant runs the same merge idea through a different tool, and its settings are not given in the card [arxiv:2308.02976 §3], so the sketch above is not a reproduction of it. The WordPiece, Galactica and byte-level variants are not reproduced by this sketch either: the cards give their vocabulary sizes and splitting rules, not their learning procedures. The byte-level learnable tokenizer learns block weights rather than merges, so its steps are in the card's pseudocode ([A New Generation of Perspective API](/papers/2202.11176)). ## Contradictions Within the first paper, the best variant depends on the metric and the language pair. Across the other papers there is no direct disagreement on numbers: they cover different languages, corpora and metrics, and none of the cards isolates its tokenizer against another's. Vocabulary sizes (30K for Dutch, 32K for Spanish, 100K for Arabic, 50K for science, 60K to 90K for NMT) are therefore not comparable, and the byte-level model has no vocabulary to compare. Naming differs: the Spanish paper calls its method BPE through SentencePiece in §3, but says WordPiece tokenization with the learned vocabulary in §5.1 [arxiv:2308.02976 §3] [arxiv:2308.02976 §5.1]. The Arabic paper calls its tokenizer WordPiece [arxiv:2101.01785 §3.1]. The text does not settle which segmentation was used at fine-tuning time in the Spanish paper. - **English-German.** Character bigrams give the best ensemble BLEU [arxiv:1508.07909 tab. 2]. Joint BPE gives the best rare-word and OOV unigram F1 [arxiv:1508.07909 tab. 2]. - **English-Russian.** Character bigrams and joint BPE tie on ensemble BLEU at 24.1. Character bigrams score higher as single models, 20.9 against 20.4 [arxiv:1508.07909 tab. 3]. Joint BPE scores higher on rare-word unigram F1, 29.7 against 27.8 [arxiv:1508.07909 tab. 3]. - **Headline gains.** The 1.1 BLEU gain for English-German comes from character bigrams, not BPE. The 1.3 BLEU gain for English-Russian is matched by character bigrams as well as joint BPE, both at 24.1 ensemble BLEU. Read each gain against the system that produced it [arxiv:1508.07909 tab. 2] [arxiv:1508.07909 tab. 3]. - **Dutch vocabulary, paper against code.** The paper builds a SentencePiece model and converts it to WordPiece [arxiv:1912.09582 §2.1]. The released script trains a unigram SentencePiece model with 30,001 pieces. The card also notes that the Hugging Face vocabulary changed in 2021, so the paper-era vocabulary is one specific revision. Both sides are stated; the card does not settle which one the released weights used. - **Dutch masking, paper against code.** The paper masks all wordpieces of a word together [arxiv:1912.09582 §2.2]. The released code turns whole-word masking off, so each wordpiece is a separate candidate. Both sides are stated; check which one you mean before you train. ## Papers - [Neural Machine Translation of Rare Words with Subword Units](/papers/1508.07909): introduces BPE for learned subword segmentation in NMT, and compares it with character bigrams and word-level baselines with dictionary back-off. - [BERTje: A Dutch BERT Model](/papers/1912.09582): uses a 30K WordPiece vocabulary, derived from SentencePiece, for a Dutch-only BERT-base encoder. - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785): uses a 100K WordPiece vocabulary for two BERT-Base encoders, one trained on formal Arabic and one on tweets. - [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](/papers/2202.11176): uses UTF-8 bytes and a learned Charformer-style tokenizer, with no fixed vocabulary, for a multilingual toxicity encoder. - [Galactica: A Large Language Model for Science](/papers/2211.09085): uses a 50K BPE vocabulary, with character-level splitting of digits, operators and sequence blocks, for a scientific language model. - [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976): uses a 32K BPE subword vocabulary, built with SentencePiece, for a Spanish-only BERT-style encoder.