GottBERT: a pure German Language Model

In plain words

Most BERT-style models used for German were trained on English, or on many languages at once. GottBERT is a model trained only on German text. The authors took the German part of OSCAR, a large corpus cut from web pages, and trained a RoBERTa-style model on it. The model learns by hiding words and guessing them back, and what it learns can then be reused for other tasks.

They also asked whether cleaning the web text helps. They removed lines with broken character encodings, junk pages and pages that are not German, using simple rules plus a small classifier. The cleaned corpus is 121GB against 145GB for the raw one.

On six German benchmarks, the base-sized GottBERT models came out ahead of the other base-sized models on four, according to the authors. The large models did not come first on any of them. Filtering made no clear difference.

TL;DR

GottBERT is a German-only RoBERTa encoder in base and large sizes. It was pretrained with fairseq on German OSCAR (145GB) and on a 121GB version filtered with rules and a one-class SVM §3. Pretraining runs 100k updates with batch 8k and peak learning rates of 4e-4 (base) and 1.5e-4 (large) §3. The authors report that base models lead on 4 of 6 tasks tab. 3. The margins are small: the best-checkpoint pairs of filtered and unfiltered models differ by at most 0.85 points, and no seed variance or significance test is reported.

Problem

BERT-style encoders remain useful because they are cheaper to run than large prompted LLMs §1. Multilingual models split capacity across languages, and the authors argue that single-language models can beat them with less pretraining effort and data §1. The German single-language models the paper lists are BERT-style (GermanBERT, dbmdz BERT, GBERT) or ELECTRA-style (GELECTRA) §2 tab. 2. The authors state that GottBERT is the first German single-language RoBERTa model §1 §6. They also ask whether filtering the German OSCAR text changes the result; the idea is inspired by FlauBERT §1.

Method

GottBERT is a RoBERTa-base or RoBERTa-large encoder §3, Pre-training. Two corpus versions were used, the raw German OSCAR portion and a filtered one, which gives four pretrained models: GottBERT base and large on unfiltered text, and GottBERT-f base and large on filtered text §3. The "best" and "last" checkpoints are kept, which gives seven checkpoints in total, since the large unfiltered model has one checkpoint for both §4.

Corpus

• Source: the German portion of the first published deduplicated OSCAR. It is 145GB, about 21.5 billion words, in about 459 million documents, one document per line §3, Training Data. The paper names no other version or URL.
• Problems the authors found: umlauts garbled by a UTF-8 text decoded as another encoding tab. 1 app. A; meaningless documents such as word lists, spam, ASCII art fig. 1 and source code app. A; and non-German text [§3, Filtering OSCAR].
• Filtering rules, applied in this order §3, Filtering OSCAR:
  1. drop any line that contains an unrecoverable umlaut marker;
  2. run clean-text on the remaining documents, configured to remove phone numbers, e-mail addresses, URLs and emoji;
  3. keep only documents of at least 40 characters;
  4. run n-gram language detection (whatlang), which also removes ASCII art;
  5. run a single-class SVM that flags meaningless documents.
• SVM: trained without labels on 12k documents. Its one hyperparameter, nu, was grid-searched and checked on 1750 annotated documents §3, Filtering OSCAR. The best model reached weighted F1 0.8578 and MCC 0.5663 §3.
• Result: 121GB, about 18.1 billion words, in about 382 million documents §3.

The paper also defines four per-document ratios: stopwords, punctuation, unique words and upper-case tokens. It does not say whether they feed the SVM or set a threshold. The definitions, as printed in §3, Filtering OSCAR (unnumbered display equations), are:

r_s = \frac{\sum_{i}^{n}\sum_{t_s \in S}[t_i = t_s]}{n}, \quad S \text{ is the nltk stopword list}

r_p = \frac{|\{\,t_i\,|\, t_i \notin W(D)\,\}|}{n}

r_u = \frac{|\{\,w_i\,|\, \forall w_j \in W(D): w_i \neq w_j\,\}|}{n}

r_{up} = \frac{\sum_{w_i \in W(D)} c(w_i)}{n}, \quad c(w) = \begin{cases} 1 & \text{if } w \text{ is capital} \\ 0 & \text{otherwise} \end{cases}

Here W(D) is the list of word tokens of document D and n its token count. The formula for r_u is ill-defined as printed; see Pitfalls.

Tokenizer

• A byte-pair subword vocabulary of 52k units, learned on a random 40GB sample of German OSCAR §3, Pre-processing. The vocabulary has 52,009 entries in the model table tab. 4.
• No language-specific pre-tokenizer such as Moses is used §3, Pre-processing.
• Compared with the English GPT-2 tokenizer, the binary data fed to fairseq is about 40% smaller §3, Pre-processing.

Pretraining

# Filter the raw German OSCAR text  [§3, Filtering OSCAR] [app. A]
# Input:  documents (one per line); svm fitted on 12k unlabeled documents
# Output: documents that pass every rule
def filter_corpus(documents, svm):
    kept = []
    for document in documents:
        # Drop any line with an unrecoverable umlaut marker  [§3] [tab. 1]
        if has_unrecoverable_umlaut(document):
            continue
        # Repair encodings; strip phone numbers, e-mails, URLs, emoji  [§3]
        document = clean_text(document)
        # Keep documents of at least 40 characters  [§3]
        if length(document) < 40:
            continue
        # Language check; the threshold is not stated  [§3]
        if detect_language(document) != "de":
            continue
        # One-class SVM flags meaningless text; its input is not stated  [§3]
        if svm.is_outlier(document_features(document)):
            continue
        kept.append(document)
    return kept

# Pretrain a RoBERTa-style masked LM with fairseq  [§3, Pre-training]
# Input:  corpus (one document per line); architecture "base" or "large"
# Output: one checkpoint saved after each epoch
def pretrain(corpus, architecture):
    # Learn 52k subword units on a 40 GB sample  [§3, Pre-processing]
    tokenizer = learn_subwords(sample(corpus, gigabytes=40), size=52_000)
    # Feed the corpus as one stream, without document boundaries  [§5]
    stream = tokenizer.encode_stream(corpus)
    model = roberta(architecture)  # RoBERTa base or large  [§3]
    # Peak learning rate: 4e-4 for base, 1.5e-4 for large  [§3]
    peak_lr = 4e-4 if architecture == "base" else 1.5e-4
    # 10k warmup steps, then polynomial decay to zero, 100k updates  [§3]
    schedule = warmup_then_polynomial(peak_lr, warmup=10_000, total=100_000)
    for step in range(100_000):
        # Batch size 8k; the unit (sequences or tokens) is not stated  [§3]
        batch = next_batch(stream, size=8_000)
        # Masking rate is not stated  [§3]
        loss = masked_lm_loss(model, batch)
        # Trained in 32-bit mode  [§5]
        update(model, loss, schedule(step))
        # Evaluate and save a checkpoint at the end of each epoch  [§4]
        if end_of_epoch(step):
            save_checkpoint(model, step)
    return model

• Hardware: the unfiltered base model ran on a 256-core TPUv3 pod, and the other models on a 128-core TPUv4 pod §3, Pre-training. Training used 32-bit mode because 16-bit was not implemented or tested in the fairseq version used §5.
• Cost: about 1.2 days for base and about 5.7 days for large §3, Pre-training.
• Checkpoints: the base "best" checkpoint was saved after 91,848 steps (12 epochs). The filtered base and large "best" checkpoints were saved after 94,530 steps (15 epochs). The unfiltered models reached up to 13.07 epochs and the filtered ones up to 15.87 §4.

Downstream tasks and fine-tuning

# Fine-tune per task and report one test score  [§3, Downstream Tasks] [tab. 5]
# Input:  pretrained model; train, dev and test splits of one task
# Output: the test score of the run chosen on dev
def evaluate_task(model, task):
    runs = []
    # 6 learning rates x 4 batch sizes = 24 runs  [tab. 5] [§3]
    for learning_rate in [5e-5, 2e-5, 1e-5, 7e-6, 5e-6, 1e-6]:
        for batch_size in [16, 32, 48, 64]:
            # Max 30 epochs for NER and classification, 10 for NLI  [§3]
            run = finetune(model, task, learning_rate, batch_size)
            # Keep the epoch with the best dev score  [§3]
            run.best = max(run.epochs, key=dev_score)
            runs.append(run)
    best = max(runs, key=lambda run: dev_score(run.best))
    # Report the test score of the run chosen on dev  [§3] [tab. 3]
    return test_score(best)

Tasks, with the splits the paper uses §3, Downstream Tasks:

• NLI: German XNLI, with 122k training, 2,490 dev and 5,010 test examples. Metric: accuracy.
• NER, CoNLL 2003 (German): four entity classes. Metric: F1.
• NER, GermEval 2014: finer labels and nested entities, scored with an adapted F1 that also credits the class hierarchy §3, Named Entity Recognition.
• Text classification, GermEval 2018: coarse (binary) and fine (four classes) tasks on tweets. The original training set is split 54% train, 6% dev and 40% test. Metric: mean of per-class F1.
• Text classification, 10kGNAD: about 10k news articles from an Austrian newspaper in nine categories. The paper's split is 90% train and 10% test, with 10% of the training set held out as dev. Metric: mean of per-class F1 §3, Text Classification (see Pitfalls for the accuracy wording in §4).

The NLI hyperparameters are described as those of the Facebook/fairseq implementation, adopted where Hugging Face offers them §3, Downstream Tasks. Fine-tuning used the Hugging Face scripts with transformers v4.34.1 on Nvidia Titan RTX and A40 GPUs §3 app. D.

Hyperparameters

Per-task fine-tuning settings selected on dev, in the paper's column order tab. 6. Each cell gives batch size (BF) and learning rate (LR):

NLI is not in tab. 6. Its settings are the ones described for NLI in §3.

Results

Author-reported. Every number is the test score of the best of 24 runs, with the run chosen on dev tab. 3 caption. Baselines are described in tab. 2: GBERT and GELECTRA (trained on OSCAR plus other sources, 163.4GB in total), dbmdz BERT (16GB), GermanBERT (12GB), mBERT (104 languages, data size not stated) and XLM-R base and large (100 languages, 2.5TB). The table below is tab. 3. Metrics: NLI accuracy; all other columns F1, per the caption. The 10kGNAD column's metric is disputed (see Pitfalls).

Headline results, from tab. 3 and §4:

• Base models: GottBERT is best on four of six tasks. These are GermEval 2014 (87.59, GottBERT-f base last), CoNLL 03 (86.14, GottBERT-f base best), GermEval 2018 coarse (78.65, GottBERT-f base best) and GermEval 2018 fine (53.92, GottBERT base last).
• Base models, not best: NLI goes to GELECTRA base (81.70), and 10kGNAD goes to dbmdzBERT (90.34).
• Large models: no GottBERT model is best on any task. GELECTRA large leads NLI (86.33), coarse (81.28) and 10kGNAD (90.97). XLM-R large leads GermEval 2014 (88.83). GBERT large leads CoNLL 03 (87.19) and fine (57.37).
• Filtering: the best-checkpoint rows of filtered and unfiltered models differ by at most 0.85 points on every task. The "last" pairs differ more on one base task: 53.92 against 52.39 on GermEval 2018 fine. The authors report no significant effect §4 §5 §6.
• Compute: the downstream runs took 113 to 673 hours per task, 1549 h 29 min in total tab. 7.

Paper vs code

No official code.

What was searched: the arXiv abstract page (no code, project or repository link); the paper text, which says the models are released in Hugging Face and fairseq format §6 and gives no URL; and a GitHub repository search for "gottbert", "scheible" and "pure German language model", which returned no matching repositories. The released model weights are not a code repository.

Pitfalls

• 10kGNAD metric is inconsistent. The Method says mean per-class F1 §3, Text Classification. The Results text calls the same numbers accuracy §4. The tab. 3 caption says all metrics except NLI are F1. Confirm the metric on the implementation side before comparing numbers.
• Selection wording is ambiguous. The text says the score is "the best of 24 runs" with "best score selection based on the validation set" §3. Read this as: pick the run by dev score, then report its test score. Reporting the best test score across all runs would be a different and optimistic protocol.
• NLI epochs conflict. The text sets 10 epochs for NLI §3, while tab. 5 lists 30 epochs for the whole grid. It is also unclear whether NLI used the 24-run grid or only the Facebook settings §3.
• Text disagrees with the table. The NER paragraph gives 86.78 CoNLL F1 for the filtered large "last" model §4. tab. 3 gives 86.28 for that row; 86.78 is the unfiltered large model and GELECTRA large.
• Filtering counts do not add up. The SVM evaluation reports 334 dirty and 1466 clean documents, which is 1800 in total, not the 1750 stated [§3, Filtering OSCAR]. The stated shares, 15.5% and 81.5%, sum to 97%.
• The r_u formula is ill-defined as printed. The set of words w_i with w_i \neq w_j for all j includes j = i, so the intended quantity is unclear. r_{up} sums over word tokens W(D) but divides by the token count n. The paper does not state the thresholds or whether the ratios feed the SVM.
• Hyperparameter CSV labels are swapped. In the source, the CSV headers list learning rate before batch size, but the values follow the rendered table's batch-then-LR order. tab. 6 uses batch then LR, as read above.
• Table and equation numbering. The appendix refers to "Table 3" for computation time, but that number is the results table. Compute times are in tab. 7. Display equations are unnumbered, and the subsections of §3 are unnumbered in the compiled paper, so cite them by name.
• "Dirty" means unfiltered. The Results text calls the unfiltered models "dirty" §4. The rest of the paper uses "unfiltered".
• Checkpoint steps are unclear. Pretraining is set to 100k updates §3. The best checkpoints are at 91,848 steps (base) and 94,530 steps (filtered) §4. The step count of the "last" checkpoints is not stated. The Results text says the filtered models finished "approximately 1 epoch earlier", while the Discussion says they were trained "for an additional epoch" §5.
• Not stated: optimizer settings, weight decay, dropout, maximum sequence length, masking rate, batch unit, and tokenizer algorithm details. Whole-word masking is listed as a future measure §5 §6, which suggests it was not used, but the paper does not say so. Do not fill these gaps with RoBERTa defaults unless you label them as your assumptions.
• Single runs and no variance. The paper reports no repeated pretraining runs, no seed variance and no significance test, even though it cites seed sensitivity §5. Gaps under about one point, such as the filtering comparison, are within unknown noise.
• Compute scope. tab. 7 gives total downstream time but does not say whether that covers one model or all of them. Pretraining hardware differs between the unfiltered base model (TPUv3) and the others (TPUv4) §3.
• Corpus version. The paper names only "the first published deduplicated version of OSCAR" and gives no version tag or URL. Reproducing the corpus depends on that exact release.
• Perplexity curves. Some pretraining runs show upward spikes, which the authors say could be read as divergence fig. 2 app. C.
• Model license. The paper says the models are released under MIT §1 §6. No hub page is linked, so this has not been checked against a release.
• Baseline data. The data size for mBERT is unknown tab. 2. The paper does not say whether the baselines were run under the same 24-run protocol §3.

Lineage

• Builds on: RoBERTa (Liu et al., 2019) and BERT (Devlin et al., 2019), which the architecture and pretraining follow §1 §3. The paper gives no arXiv id, and neither has a card in this wiki.
• Inspired by: FlauBERT (Le et al., 2020), whose filtering of a large corpus the authors adapt §1.
• Compared against: GBERT and GELECTRA (Chan et al., 2020, no arXiv id given), dbmdz BERT, GermanBERT, mBERT and XLM-R tab. 2 tab. 3. None has a card in this wiki.
• Related card: CamemBERT, a French RoBERTa. The authors cite it for the argument that a more varied corpus helps §5.
• Follow-ups named in the paper: BioGottBERT, a medical variant §5.
• Superseded by: not known from the content in this wiki.