BERTje: A Dutch BERT Model

In plain words

Most language models that understand text were trained mainly on English. A model trained on many languages at once, such as multilingual BERT, covers Dutch, but for Dutch it learned mostly from Wikipedia. BERTje is a model of the same size and design, trained only on Dutch text from several sources: fiction books, news, Wikipedia and a reference corpus of written Dutch.

The training changes the original BERT recipe in two places. First, the model is trained to tell whether two sentences are in their real order or have been swapped, instead of whether the second sentence follows the first. Second, it hides whole words, not single fragments of words, when it asks the model to fill in blanks.

After training, the model is adapted to specific tasks such as finding names of people and places, tagging word classes, labelling who did what in a sentence, and judging whether a book review is positive or negative. On each of these tasks it scores higher than multilingual BERT. The results are the authors' own, and the code they released is research code, not a production-ready package.

TL;DR

BERTje is a BERT-base model trained only on Dutch text: about 2.4 billion tokens from books, news, Wikipedia and SoNaR §2.1. It replaces next sentence prediction with sentence order prediction and masks 15% of tokens §2.2. The authors report that it beats multilingual BERT on every test column of their tables, for NER, POS, semantic roles, spatio-temporal relations and sentiment tab. 1 tab. 2 tab. 3 tab. 4. The released code differs from the paper in masking, data splits and evaluation; see Paper vs code before implementing.

Problem

Multilingual BERT covers Dutch, but its Dutch part is trained on Wikipedia only, and Wikipedia is not representative of general language use §1. A monolingual model may do better on a single language §1. Monolingual BERT-style models exist for Italian, French, German, Finnish and Japanese §1. A Dutch model available at the time (textdata.nl) was consistently beaten by multilingual BERT in the authors' experiments [§1, footnote]. The paper's aim is a high-performing Dutch model that is compared with multilingual BERT on the same tasks §1.

The paper also argues that next sentence prediction (NSP) is ineffective, because the model can learn topic similarity instead of coherence §1. It cites RoBERTa for removing NSP and ALBERT for replacing it with sentence order prediction (SOP) §1.

Method

Architecture

The model is architecturally equivalent to BERT-base, with 12 transformer blocks §2. The paper does not state the hidden size, head count or feed-forward size beyond this equivalence. The architecture config file is referenced in the pretraining script but is not in the repository run-pretraining.sh.

Pre-training data

Five corpora, with sizes as uncompressed text after cleaning §2.1:

• Books: contemporary and historical fiction novels, 4.4 GB.
• TwNC, a Dutch news corpus, 2.4 GB.
• SoNaR-500, a multi-genre reference corpus, 2.2 GB.
• Web news: all articles from four Dutch news websites, 1 January 2015 to 1 October 2019, 1.6 GB.
• Wikipedia: the October 2019 dump, 1.5 GB.

Before use, the authors removed chat and Twitter documents from SoNaR, removed the Wikipedia documents from SoNaR, and removed SoNaR-500 documents that also appear in SoNaR-1 and Lassy Small §2.1. The final corpus is 12 GB of uncompressed text, about 2.4B tokens §2.1. The Books and Web news sources are not named further in the paper, so the corpus cannot be rebuilt from the paper alone.

Vocabulary

The paper uses a WordPiece vocabulary of 30K tokens, created by training a SentencePiece model on the raw pretraining data and converting it to WordPiece format §2.1. The released script trains a unigram SentencePiece model with 30,001 pieces, on a sample of up to 100 million sentences, with [CLS], [SEP], [UNK], [PAD] and [MASK] as control or special symbols create-vocab.py [#L19]. A second script converts the output to WordPiece format vocab.py.

Pre-training objectives

Each training example is a pair of text segments. Two objectives are trained together:

• Masked language modeling (MLM): 15% of tokens are selected. Of the selected tokens, 80% become a mask token, 10% become a random token and 10% are left unchanged §2.2. The paper says that the masking hides consecutive wordpieces that belong to the same word, not single wordpieces §2.2. The released code does not do this; see Paper vs code.
• Sentence order prediction (SOP): the second segment is either the next or the previous sentence §2.2. This replaces NSP §2.2.

No numbered equations and no algorithm box appear in the paper. The pseudocode below is written from the paper's description and the released code.

# Build one sentence-order pair from a document  [§2.2]
# Input:  document, list of sentences (each a list of wordpiece tokens)
# Output: tokens_a, tokens_b, is_swapped
def make_sop_pair(document, target_length, max_num_tokens, rng):
    # Take consecutive sentences until they reach the target length
    # [code: pretraining/create-pretrain-data.py#L218-L222]
    chunk = take_until_length(document, target_length)
    # Split the chunk into sentence A and sentence B at a random point
    # [code: pretraining/create-pretrain-data.py#L226]
    cut = rng.randint(1, len(chunk) - 1)
    tokens_a = flatten(chunk[:cut])
    tokens_b = flatten(chunk[cut:])
    # Swap A and B half of the time, so B comes before A  [§2.2]
    # [code: pretraining/create-pretrain-data.py#L237-L240]
    is_swapped = rng.random() < 0.5
    if is_swapped:
        tokens_a, tokens_b = tokens_b, tokens_a
    # Trim the longer side, from the front or the back, until the pair fits
    # [code: pretraining/create-pretrain-data.py#L362-L377]
    truncate_pair(tokens_a, tokens_b, max_num_tokens, rng)
    return tokens_a, tokens_b, is_swapped

# Choose and corrupt the masked positions  [§2.2]
# Input:  tokens (list of str, including [CLS] and [SEP])
# Output: corrupted tokens, positions, original tokens at those positions
def mask_tokens(tokens, vocab_words, rng):
    # Budget: 15% of tokens, at least 1, at most 20 per sequence
    # [§2.2] [code: pretraining/create-pretrain-data.py#L313-L314]
    budget = min(20, max(1, round(len(tokens) * 0.15)))
    # Group the wordpieces of each word; the paper masks whole words
    # [§2.2] [code: pretraining/create-pretrain-data.py#L290-L307]
    groups = group_wordpieces(tokens)
    shuffle(groups, rng)
    chosen = []
    for group in groups:
        # Skip a group that would exceed the budget
        # [code: pretraining/create-pretrain-data.py#L323-L324]
        if len(chosen) + len(group) > budget:
            continue
        chosen.extend(group)
        if len(chosen) == budget:
            break
    corrupted = list(tokens)
    for index in chosen:
        # 80% of chosen tokens become [MASK]  [§2.2]
        # [code: pretraining/create-pretrain-data.py#L337-L338]
        if rng.random() < 0.8:
            corrupted[index] = "[MASK]"
        # Half of the other 20% keep the original token  [§2.2]
        # [code: pretraining/create-pretrain-data.py#L341-L342]
        elif rng.random() < 0.5:
            pass
        # The other half get a random vocabulary token  [§2.2]
        # [code: pretraining/create-pretrain-data.py#L343-L345]
        else:
            corrupted[index] = random_choice(vocab_words, rng)
    positions = sorted(chosen)
    return corrupted, positions, [tokens[i] for i in positions]

Pre-training schedule

The model is pretrained for 1 million iterations, and the authors also evaluate a 850k-iteration checkpoint §2.2. The paper gives no batch size, learning rate, warmup or sequence length. The shell command in the repository sets them as listed under Hyperparameters run-pretraining.sh. The command calls a run-pretraining.py script that is not in the repository. Pretraining used Cloud TPUs [Acknowledgments].

Fine-tuning

Each task is fine-tuned from the pretrained model with the same hyperparameters, for four epochs. The authors say that longer training degraded validation performance on some data and that no improvement appeared after the fourth epoch §4. The paper does not state the batch size, learning rate or optimizer; the released scripts set them as listed below. The paper does not name the script version it used. The v1 scripts match its stated four-epoch protocol, while the v2 configuration does not (see Paper vs code).

# Fine-tune one task, same recipe for each task  [§4]
# Input:  pretrained BERTje, train_set
# Output: model after the last epoch, which is the one evaluated
def fine_tune(model, train_set):
    # AdamW, learning rate 5e-5, eps 1e-8; no scheduler appears in the script
    # [code: finetuning/v1/run_benchmark.py#L122] [#L138]
    optimizer = adamw(model.parameters(), lr=5e-5, eps=1e-8)
    for epoch in range(4):  # four epochs  [§4] [code: #L122]
        for batch in shuffled_batches(train_set, size=8):
            # Loss is taken at every position, padding included
            # [code: finetuning/v1/run_benchmark.py#L157-L160]
            loss = model.loss(batch.inputs, batch.labels)
            loss.backward()
            # Clip the gradient norm to 1.0 (the sentiment script does not)
            # [code: finetuning/v1/run_benchmark.py#L164]
            clip_grad_norm(model.parameters(), 1.0)
            optimizer.step()
            model.zero_grad()
    return model

Tasks and data

• Named-entity recognition on the Dutch CoNLL-2002 data, four classes: person, organisation, location, miscellaneous. Standard train, dev and test splits §3.
• POS tagging with the 16 universal tags on the Lassy Small treebank in Universal Dependencies v2.5 §3.
• Tasks from SoNaR-1: token-level NER (6 labels), coarse POS (12 labels), fine-grained POS (241 labels, 223 present in training), semantic-role predicate-argument labels, semantic-role modifier labels and spatio-temporal relations. Each annotation is flattened to token-level classes. The paper says the split is 80% train, 10% validation and 10% test, at document level §3. The released script uses a different split; see Paper vs code.
• Sentiment analysis on the 110k Dutch Book Reviews Dataset: a balanced binary task §3.

Hyperparameters

Results

Author-reported. Setup: BERTje (BERT-base size) and multilingual BERT base, each fine-tuned for four epochs on each task §4. Number of runs and variance: not stated. Single numbers in each cell. "850k" is the 850k-iteration checkpoint; "BERTje" is the 1M-iteration model.

Named-entity recognition, F1 tab. 1. Span-level F1 with the CoNLL-2002 script.

The paper says BERTje does not beat the 90.9 test score of Wu and Dredze on CoNLL-2002. That score comes from a large multilingual model, which the authors describe as well-optimized §4.

Part-of-speech tagging, accuracy tab. 2.

On UD-LassySmall, the paper compares with UDPipe 2.0 at 95.98% accuracy. It notes that UDPipe was evaluated on UD 2.2, while this paper uses UD 2.5, and calls the difference minor §4.

Semantic roles and spatio-temporal relations, F1 tab. 3. Semantic-role scores use the CoNLL-2002 script, and STR is a macro F1.

The paper states that these results are stand-alone, because the authors know of no similar systems for this data §4.

Sentiment analysis, accuracy tab. 4. Train and test, on the 110k Dutch Book Reviews Dataset.

The paper notes that ULMFiT's score came from manual hyperparameter tuning and that BERTje, without tuning, comes close to it §4.

The paper observes that the 850k checkpoint matches the full model on POS, but the full model gains on NER, which the authors take as a sign that higher-level structure is learned later in pretraining §4 §5.

Paper vs code

Code: github.com/wietsedv/bertje, commit 1f2cea78e9242f98575ca34f0b61f34bf3ab3b24, Apache-2.0. The owner name matches the first author, and the paper's abstract links the repository. The released code covers pretraining data preparation, vocabulary creation, fine-tuning and probing. The README says it is research code and not complete README.md.

1. Masking unit. Paper: consecutive wordpieces of the same word are masked together §2.2. Code: WHOLE_WORD_MASK = False, so each wordpiece is a separate candidate create-pretrain-data.py [#L303-L307].
2. Masking budget. Paper: 15% of tokens §2.2. Code: at most 20 predicted tokens per sequence, with 15% as the target before the cap create-pretrain-data.py [#L313-L314] and the same cap in the run command run-pretraining.sh. For a full 509-token text, the cap gives about 4%, not 15%.
3. Blacklist for pretraining. Paper: SoNaR-500 documents that also appear in SoNaR-1 or Lassy Small are removed §2.1. Code: the blacklist comes from Lassy Small suites only prepare-sonar.py [#L59]. No SoNaR-1 blacklist appears in the pretraining scripts.
4. SoNaR split. Paper: 80 / 10 / 10 at document level §3. Code: dev is 10%, test is 15%, train is the rest (75%) prepare-sonar.py [#L258-L260].
5. Sentiment hyperparameters. Paper: same hyperparameters for all tasks §4. Code: sentiment uses learning rate 2e-5 and no gradient clipping run_110kDBRD.py [#L117], while the other tasks use 5e-5 with clipping at 1.0 run_benchmark.py [#L164].
6. Evaluation metric. Paper: span-based F1 with the CoNLL-2002 script for NER and SRL tab. 1 tab. 3. Code: the v1 benchmark script prints token-level sklearn reports run_benchmark.py. The v1 directory has a script to merge true and predicted labels for conlleval merge-outputs.py; the paper's span numbers may have come from that step, but this cannot be verified from the repository.
7. Sentiment test set. The sentiment script has no separate dev set. The per-epoch "Test Acc" is measured on the test set run_110kDBRD.py [#L187]. The paper reports test accuracy only tab. 4.
8. Sentence-order label. The released code uses label 1 for an in-order pair and 0 for a swapped pair create-pretrain-data.py. Check this against any NSP-style label convention you reuse.
9. Later procedure. The README reports benchmark numbers from after the paper. They use the same procedure for each model, and they differ from the paper's tables (CoNLL-2002 test for BERTje: 90.24 in the README, 88.3 in the paper) README.md. The v2 configuration trains up to 80 epochs with batch 6, gradient accumulation of 4, 512 warmup steps and dropout 0.2/0.3 default.yaml [#L65-L66], which does not match the paper's four epochs.
10. Pretraining script. The shell command calls run-pretraining.py, which is not in the repository run-pretraining.sh. The optimizer type, weight decay and learning-rate schedule cannot be checked from the repository.
11. Vocabulary changed. The README says the vocabulary of the Hugging Face model changed in 2021. The paper-era vocabulary is revision v1 of GroNLP/bert-base-dutch-cased README.md.
12. Dataset not in the repository. The repository has preparation scripts for Books, TwNC, SoNaR and Web news, but not the data. The Books source is not named in the paper.

Pitfalls

• Masking is much lighter than the paper says. With the released settings (cap of 20 per sequence, no whole-word grouping), the masked fraction is far below 15% for full-length inputs. An implementation that follows the paper's 15% and whole-word rule will not match the released model's training. Decide which one you mean before you train.
• Trailing text is dropped. In the pair builder, the loop stops one sentence before the end of each document, so the last sentence of a document is never used. A partial chunk at the end of a document, shorter than the target length, is discarded too create-pretrain-data.py.
• Fine-tuning loss includes padding and subword positions. The loss is computed over every position, while accuracy uses the label mask. Subword continuations are labelled O run_benchmark.py [#L176]. Mask the loss yourself if you want only word-initial tokens.
• Attention dropout may not be applied. The script passes attention_dropout_prob, but the standard transformers field for attention dropout is attention_probs_dropout_prob run_benchmark.py. If the library ignores the unknown name, attention dropout stays at its default. This was not run here.
• Per-token encoding may add special tokens. Each word is encoded with tokenizer.encode(token), and the first label is put on the first returned id run_benchmark.py. If the library version adds [CLS] and [SEP] to each word, the first label lands on [CLS]. Check this in your library version; it was not verified here.
• Long inputs are cut at 512 wordpieces. Labels are cut with them run_benchmark.py.
• Casing is not set for NER and the other token tasks. The v1 script calls BertTokenizer.from_pretrained with no do_lower_case, so the model's own tokenizer config decides run_benchmark.py. The sentiment script sets do_lower_case=False run_110kDBRD.py.
• Compute. 1,000,000 steps at batch 256 is about 256 million sequences of up to 512 tokens, run on 8 TPU cores run-pretraining.sh [#L20-L22]. The paper does not state the cost.
• Duplication and short sequences are undocumented. Each document is used 10 times with different masks and pairs, and 10% of examples are shorter on purpose create-pretrain-data.py [#L38] [#L206-L207]. The paper mentions neither.
• Single runs, no variance. The paper reports one number per cell and no seed study, so differences of about one point are not established.
• Results are for one language and one size. The paper tests Dutch only and one base-size model; it does not show that the approach transfers to other languages.

Lineage

• Builds on: BERT, a transformer pretrained with masked language modeling and next sentence prediction (Devlin et al., 2019, NAACL, doi:10.18653/v1/N19-1423). BERTje keeps the BERT-base architecture and changes the data, the masking rule and the sentence-pair task §2.
• Changes from RoBERTa and ALBERT: the paper cites RoBERTa for dropping NSP (arXiv:1907.11692) and ALBERT for replacing it with sentence order prediction (arXiv:1909.11942). BERTje uses SOP §1 §2.2.
• Compared against: multilingual BERT tab. 1-tab. 4; the state-of-the-art CoNLL-2002 score of Wu and Dredze (2019) §4; ULMFiT as evaluated by van der Burgh and Verberne (2019), arXiv:1910.00896 §4; and UDPipe 2.0 §4.
• Superseded by: not known from this wiki.