CamemBERT: a Tasty French Language Model

In plain words

Big pretrained language models like BERT and RoBERTa were, at the time, mostly available for English, or as "multilingual" models that split their capacity across about a hundred languages. If you worked on French text, you either used an English-centric model or a multilingual one that was larger and often worse at French specifically.

CamemBERT is simply RoBERTa retrained from scratch on French. The authors took French text scraped from the web (the OSCAR corpus), cut it into subword pieces with SentencePiece, and trained the model to fill in hidden words. When a word is split into several pieces, they hide all of its pieces together, so the model cannot cheat by looking at the rest of the word.

The resulting model did better than the multilingual competitors on French grammar tagging, syntactic parsing, named entity recognition and sentence-pair reasoning. The more surprising part is the data study: a random 4GB slice of web text worked about as well as the full 138GB, and messy, varied web text worked better than clean Wikipedia text of the same size. That suggests a language with only a few gigabytes of web text can get a strong model of its own.

TL;DR

A RoBERTa-BASE (and one LARGE) encoder trained only on French, with a 32k SentencePiece vocabulary and whole-word masking, using fairseq. Pretraining is 100k steps at batch 8192 x 512 tokens, peak LR 7e-4, on 256 V100s for about a day. Fine-tuned, it is reported as state of the art on four French UD treebanks (UPOS/LAS), FTB NER and French XNLI. The ablations are the reusable part: Common Crawl beats Wikipedia at equal size, 4GB of diverse web text is close to 138GB, whole-word masking helps NLI slightly but not the low-level tasks, and LARGE helps NLI/NER but not tagging/parsing.

Problem

Pretraining needs hundreds of gigabytes of text and hundreds of GPUs, so strong monolingual models existed mostly for English; multilingual models such as mBERT or XLM-R are larger and, for a high-resource language like French, can trail a monolingual model §1. The paper asks whether a RoBERTa-style model can be trained for another language (French) and how much and what kind of pretraining data it needs §1 §6.

Method

CamemBERT reuses RoBERTa's recipe with two changes: SentencePiece tokenization instead of WordPiece, and whole-word masking §4.3.

Data. French part of OSCAR, a Common Crawl corpus filtered only by a fastText language classifier (176 languages), with no further filtering. The non-shuffled version is used: 138GB of raw text, 32.7B tokens after subword tokenization §4.1. Alternative corpora for the ablations: French Wikipedia 2019 dump (cleaned with WikiExtractor, 4GB) and CCNet (the head split, top 33% of documents by filtering perplexity, 135GB) §6. Corpus statistics are in tab. 6.

Tokenization. SentencePiece, 32k vocabulary, learned on 10^7 sentences sampled randomly from the pretraining data; no subword regularisation §4.2.

Architecture. BERT-BASE (12 layers, hidden 768, 12 heads, 110M params) or BERT-LARGE (24 layers, hidden 1024; 335M params per tab. 8) §4.3.

Objective. MLM with dynamic masking (masks resampled each time rather than fixed in preprocessing), no next-sentence prediction §4.3.

Algorithm: whole-word masking for one sequence  [§4.3]
Input: raw text line(s), SentencePiece tokenizer
1. Split the untokenized text on whitespace to define words.
2. Tokenize each word into subword tokens.
3. Sample 15% of the words uniformly at random.
4. Every subword token of a sampled word becomes a prediction target
   (overall this is close to 15% of tokens).
5. For each target token: replace with <MASK> (80%), keep unchanged (10%),
   or replace with a random token (10%).
6. Loss = cross-entropy of predicting the original tokens at target positions.
Masks are redrawn dynamically every time a sequence is used.

Sequences. At most 512 tokens per sequence; each sequence contains only complete paragraphs, which are lines of the pretraining file §4.3.

Optimisation. Adam, warmup 10k steps to peak 7e-4, then polynomial decay to zero, 100k steps, batch 8192 sequences, fairseq RoBERTa implementation, 256 V100 32GB GPUs for one day §4.3. The authors note performance was still rising when they stopped §4.3.

Downstream use §4.4:

• Fine-tuning: for sentence classification a linear layer on the last hidden state of <s>; for token labeling a linear layer on the last hidden state of the first subword of each word; for dependency parsing a biaffine graph head following Dozat and Manning; for XNLI a head of one hidden layer with a non-linearity plus a linear projection, with input dropout on both. Adam with a fixed learning rate, grid search over learning rate and batch size, best validation checkpoint among the first 30 epochs. No weight decay, warmup or discriminative fine-tuning except for NLI, which uses fairseq's RoBERTa MNLI defaults. POS/parsing/NER use Hugging Face Transformers; NLI uses fairseq.
• Feature-based embeddings: a word's vector is obtained by averaging each subword's representations over the last four Transformer layers, then averaging over the word's subwords. These feed UDPipe Future (tagging, parsing) or an LSTM+CRF (NER).

Hyperparameters

Results

Author-reported. Main CamemBERT is BASE, OSCAR 138GB, whole-word masking, 100k steps. POS/parsing on UD v2.2 test sets with gold tokenization; best of 4 runs selected on validation tab. 2. NER on FTB, best of 4 tab. 3. NLI on French XNLI test, best of 10 tab. 4.

POS tagging (UPOS) and dependency parsing (LAS) tab. 2:

NER F1 on FTB tab. 3:

French XNLI test accuracy tab. 4:

Corpus origin and size, BASE, 100k steps, validation sets, mean of 4 runs (10 for NLI) tab. 5:

Per-treebank numbers for these runs are in tab. 5.

Design-choice ablations, validation sets, POS/parsing averaged over the 4 treebanks tab. 8:

Test-set means for all six pretrained variants in both fine-tuning and embedding settings, per treebank, are in tab. 7; e.g. fine-tuned XNLI test accuracy is 81.18 (OSCAR subword), 81.92 (OSCAR whole-word), 83.43 (CCNet whole-word 500k) and 85.29 (CCNet LARGE) tab. 7.

Findings the authors draw: Common Crawl data (4GB OSCAR or CCNet) beats 4GB Wikipedia, with the largest gaps on Spoken and XNLI §6.1; 4GB OSCAR is close to 138GB except NER, where 138GB is +0.9 F1 §6.2; whole-word masking gives +0.5 XNLI and little elsewhere app. A; LARGE gives 19.7% and 23.7% error reduction on NLI and NER but no significant gain on POS/parsing app. B; 500k vs 100k steps adds +0.84 XNLI and little on POS/parsing app. D; parsing and NER plateau around 22k steps while NLI is still rising at 100k app. D fig. 1.

Paper vs code

No official code. The paper states it used the RoBERTa implementation in fairseq for pretraining and NLI and an extended Hugging Face Transformers for the other tasks §4.3 §4.4, and that checkpoints are distributed via fairseq, Hugging Face and camembert-model.fr under MIT §8; no author training repository with configs is linked, so nothing was compared.

Pitfalls

• LARGE spec is garbled. §4.3 gives LARGE as "24 layers, 1024 hidden dimensions, 12 attention heads, 110M parameters"; tab. 8 and tab. 4 say 335M. Head count for LARGE is therefore not reliably stated in the paper.
• The LARGE model is trained on CCNet, not OSCAR §5, so the 85.7 XNLI result is not "CamemBERT on OSCAR at larger scale".
• Best-of-N vs mean. Headline tables tab. 2 tab. 3 tab. 4 report the best run selected on validation out of 4 or 10; tab. 5 and tab. 8 are validation means; tab. 7 is test means. E.g. fine-tuned GSD LAS is 92.57 in tab. 2 but 92.30 (OSCAR whole-word) in tab. 7. Compare like with like.
• Internal inconsistencies in tab. 8. The CCNet BASE whole-word 100k model appears with UPOS/LAS 97.67/89.46 in the size and dataset blocks but 98.04/89.85 in the steps block. Also, tab. 5 gives OSCAR 138GB fine-tuned NER as 91.55, which tab. 8 assigns to the subword model while whole-word gets 91.44.
• Whole-word masking needs word boundaries from the raw text. Words are defined by whitespace in the untokenized input §4.3, so the subword-to-word mapping must be kept through tokenization. How it is recovered is not stated.
• Masking granularity for 80/10/10 is unstated. Under whole-word masking it is unclear whether the mask/keep/random choice is drawn per word or per subword token; the whole-word paragraph also omits the 10% random figure, which is given only in the token-level description §4.3.
• Single pretraining seed. Each pretrained model was trained once; the authors attribute OSCAR-4GB beating OSCAR-138GB on XNLI (81.88 vs 81.55) to pretraining seed §6.2. Small differences between corpora are within noise.
• Under-training. 100k steps over 4GB is about 400 epochs and validation loss was still falling §6.2; the full-corpus model was also stopped while improving §4.3. The "4GB is enough" claim is at a fixed 100k-step budget and BASE size only.
• Sequences are paragraph-aligned. Each 512-token sequence holds only whole paragraphs (lines) §4.3; how lines longer than 512 tokens are handled is not stated.
• Spoken treebank is uncased and unpunctuated, while CamemBERT was trained on cased text; the authors suggest this explains its weaker UPOS there §5.
• Fine-tuning grids are not given. Learning rates and batch sizes searched are not listed §4.4; NLI defers to fairseq's MNLI defaults.
• FTB is research-only. The NER dataset was used under a research-only agreement §3.
• Unshuffled OSCAR. The pretraining corpus is the non-shuffled French OSCAR §4.1; how 4GB subsets were drawn is "random at the document level" §6.

Lineage

• Builds on: RoBERTa (arXiv:1907.11692) and BERT (arXiv:1810.04805); changes are SentencePiece tokenization, whole-word masking and French-only data §4.3. Whole-word masking follows the updated BERT release and SpanBERT (arXiv:1907.10529) §4.3.
• Data: OSCAR (Ortiz Suárez et al. 2019) and CCNet (arXiv:1911.00359) §4.1 §6.
• Compared against: mBERT, XLM (arXiv:1901.07291), XLM-R (arXiv:1911.02116), UDify, UDPipe Future (+ mBERT + Flair), CRF and LSTM-CRF NER baselines §3.
• Related French model: FlauBERT (arXiv:1912.05372), named in §7.