FlauBERT: Unsupervised Language Model Pre-training for French

In plain words

FlauBERT is a language model for French, built the same way as BERT, the English model that started a wave of pre-trained language models. It reads a large amount of ordinary French text and learns to guess words that have been hidden from it. The trained model is then reused: a small extra layer is added on top and the whole network is tuned on one specific task, such as judging whether a product review is positive.

The authors trained two sizes on a large, mixed French corpus. They also assembled a suite of French language tasks, called FLUE, so that French models can be compared on the same tests. On those tasks their models usually beat a multilingual model that covers many languages at once. The code and the training recipe are published, but the code is licensed for non-commercial use only.

TL;DR

• FlauBERT is a BERT-style Transformer encoder, pre-trained only with masked language modelling on a 71 GB French corpus, in a base (12 layers, hidden 768) and a large (24 layers, hidden 1024) size §3.2 tab. 1.
• The authors also release FLUE, six French datasets covering classification, paraphrase, inference, parsing and word-sense disambiguation §4 tab. 2.
• Author-reported: both sizes beat mBERT on most tasks; the large model has the top score on CLS, dependency parsing and both WSD tasks §5 tab. 3, 7, 8, 9.
• Before you reproduce anything: the paper's classification head is not the one the shipped FLUE runner uses, and the large-model settings are not in the repository. See Paper vs code.

Problem

Contextual pre-trained encoders had given large gains on English NLP tasks, but most pre-trained models for other languages were multilingual, with one shared vocabulary for many languages §1 §2.2. The authors argue that a model trained only on French should do better on French tasks. At the time they wrote, they knew of no multi-task evaluation suite for French comparable to GLUE for English §2.3.

Method

Pre-training corpus §3.1 app. A

• The corpus has 24 sub-corpora from three sources: monolingual French data from the WMT19 shared tasks, French corpora from the OPUS collection, and Wikimedia projects §3.1 tab. 10.
• Before preprocessing the text is 270 GB. Common Crawl, the largest sub-corpus, is reduced from 215 GB to 43.4 GB by aggressive cleaning §3.1.
• Preprocessing removes very short sentences and repetitive or non-meaningful content such as telephone numbers and e-mail addresses, normalizes Unicode, and tokenizes with the Moses tokenizer §3.1.
• The final corpus is 71 GB, about 12.79 B tokens and about 488.78 M sentences tab. 10 app. A.
• Per-sub-corpus sizes, token counts and sentence counts are listed in tab. 10. Common Crawl is the largest at 43.4 GB tab. 10.

Architecture §3.2 tab. 1

• FlauBERT uses BERT's architecture: a multi-layer bidirectional Transformer encoder §3.2.
• Base: L = 12 blocks, hidden size H = 768, A = 12 attention heads. Large: L = 24, H = 1024, A = 16 §3.2.
• Parameter counts: 138 M (base) and 373 M (large) tab. 1.
• Vocabulary: 50K byte-pair-encoding units, learned after Moses tokenization §3.2.
• Masking: dynamic, sub-word masking tab. 1. The masking rate is not stated in the text.

Objective §3.2

• Only masked language modelling (MLM) is used. Next-sentence prediction (NSP) is dropped; the authors cite later studies that found removing NSP does not hurt, and that the original loss may have been applied with a bad input format §3.2.
• The loss is computed only at the positions chosen for masking §3.2.

# Masked-language-model step, pre-training  [§3.2]
# Input:  token_ids (B, T) from BPE over Moses-tokenized French
# Output: scalar loss over the masked positions

def mlm_step(token_ids):
    # Choose ~15% of positions as targets  [code: README.md#L303]
    targets = sample_positions(token_ids, rate=0.15)
    # Of the chosen tokens: 80% -> mask, 10% -> random token, 10% unchanged
    #   [code: train.py#L102]
    inputs = corrupt(token_ids, targets)        # (B, T)
    # Encode the corrupted sequence with the bidirectional Transformer  [§3.2]
    hidden = transformer_encoder(inputs)        # (B, T, H)
    # Predict the original token only at the chosen positions  [§3.2]
    logits = output_projection(hidden[targets])
    return cross_entropy(logits, token_ids[targets])

The paper does not state the 15% rate or the 80/10/10 split; the values above come from the released command and the argument default in the code.

Optimization §3.2

• Optimizer: Adam with beta1 = 0.9, beta2 = 0.98, epsilon = 1e-6 and weight decay 0.01, for both sizes §3.2.
• Warmup and peak learning rate: base 24k steps and 6e-4; large 30k steps and 3e-4 §3.2.
• The paper does not say how the learning rate decays after warmup. The repository uses an inverse-square-root decay (see the pseudocode below and Paper vs code).

# Learning rate of the repo's Adam schedule  [code: xlm/optim.py#L96-L100]
# Input:  update step (int); base warmup W = 24000 steps, peak 6e-4  [§3.2]
# Output: learning rate for this step

def learning_rate(step):
    # Linear warmup from the initial rate 1e-7 to the peak
    #   [code: xlm/optim.py#L97] [code: README.md#L295]
    if step < warmup_steps:
        return init_lr + step * (peak_lr - init_lr) / warmup_steps
    # After warmup, decay with 1 / sqrt(step)  [code: xlm/optim.py#L100]
    return peak_lr * sqrt(warmup_steps) / sqrt(step)

Large model stability §3.2

• Training the large model with the BERT-large or RoBERTa-large configurations diverged early §3.2.
• The authors use pre-norm attention (layer normalization before each sub-layer) and stochastic depth, with a layer-dropping rate of 0.2 §3.2.
• The paper says the rate is 0.2 "in all the experiments". It is ambiguous whether this applies to the base model too. The normalization placement for base is not stated.

Compute §3.2

• Batch: an effective batch of 8192 sequences for both sizes §3.2.
• Base: 32 Nvidia V100 GPUs for 410 hours. Large: 128 GPUs for 390 hours §3.2.
• Derived from those numbers (not stated in the paper): about 13,120 GPU-hours for base and about 49,920 GPU-hours for large.

FLUE benchmark §4 tab. 2

Source: tab. 2.

• CLS (§4.1): binary sentiment on Amazon reviews. Three-star reviews are dropped; the French portion is used. Metric: accuracy §4.1.
• PAWS-X (§4.2): paraphrase identification. The French dev and test sets are human-translated; the training set is machine-translated. Metric: accuracy §4.2.
• XNLI (§4.3): entailment, contradiction or neutral for a premise-hypothesis pair. Training data is the machine-translated French version. Metric: accuracy §4.3.
• Parsing (§4.4 and §5.4 to §5.5): the French Treebank in the SPMRL 2013 version, with a standard split. Constituency F1 and POS accuracy, plus UAS and LAS for dependency parsing §4.4 §5.5.
• Verb WSD (§4.5): FrenchSemEval, 3,199 annotations over 66 verbs. The sense inventory is a Wiktionary dump; the inventory's examples serve as training data §4.5.
• Noun WSD (§4.5): 306 sentences and 1,445 nouns, annotated with WordNet 3.0 sense keys. BabelNet keys were converted to WordNet where a mapping exists and dropped otherwise. Training data is SemCor and the WordNet gloss corpus, translated to French with a machine-translation system and aligned to the English sense keys §4.5.

Fine-tuning §5

• CLS (§5.1): the standard BERT fine-tuning process. The input is a text paired with an empty second segment. The head is dropout, linear, tanh, dropout, linear, with dropout 0.1 §5.1. Training: 30 epochs, batch 16, grid search over four learning rates (1e-5, 5e-5, 1e-6, 5e-6), with a random 20% of the training data as validation and the best checkpoint evaluated on test §5.1.
• PAWS-X (§5.2): the same setup, but the input is a sentence pair and the hyperparameter search uses the dev set, with no extra split §5.2.
• XNLI (§5.3): a sentence pair as input. The head is one dropout layer and one linear layer. Epoch count and learning rate are not stated in the text §5.3.
• Constituency parsing (§5.4): the self-attentive chart parser of Kitaev and Klein, with POS tagging as a second task. The parser's default hyperparameters for the first two settings; for the language models, the hyperparameters of Kitaev et al. (2019, multilingual), except that large uses learning rate 1e-5, batch 8 and skips training sentences longer than 100 tokens §5.4. Each setting is trained three times with different seeds and the run with the best dev F1 is kept §5.4.
• Dependency parsing (§5.5): a parser that follows Dozat (2017) and decodes trees with a maximum spanning tree. Word input is the concatenation of trained word and tag embeddings plus at most one pre-trained vector. Word and tag embeddings use dropout 0.5; the BERT representations get no dropout §5.5. Predicted tags come from the shared task §5.5.
• Verb WSD (§5.6): no training of the classifier. Each sense is represented by the mean of the contextual embeddings of its inventory examples; a test instance gets the sense with the highest cosine similarity §5.6.
• Noun WSD (§5.6): a classifier over the pre-trained model's output: six Transformer encoder layers with feed-forward size 3072 and 12 heads, then a softmax over synsets. Eight models are trained per experiment; the mean and standard deviation of single models are reported, plus an ensemble that averages the softmax outputs §5.6.

# Classification head for CLS, as the paper describes it  [§5.1]
# Input:  pooled encoder output (B, H), H = hidden size
# Output: class logits (B, 2)
# The shipped FLUE runner uses only dropout and one linear layer
#   [code: xlm/evaluation/flue.py#L106-L110]

def classify(pooled):
    # Dropout with rate 0.1  [§5.1]
    hidden = dropout(pooled, rate=0.1)          # (B, H)
    # Linear layer, output size H  [§5.1]
    hidden = linear(hidden)                     # (B, H)
    # tanh activation  [§5.1]
    hidden = tanh(hidden)                       # (B, H)
    # Dropout again  [§5.1]
    hidden = dropout(hidden, rate=0.1)          # (B, H)
    # Final linear layer to the classes  [§5.1]
    return linear(hidden)                       # (B, 2)

The paper does not say which token is pooled; it only refers to the standard BERT fine-tuning process §5.1.

Hyperparameters

Results

Author-reported. Setup: French test sets from FLUE, fine-tuned FlauBERT base and large against mBERT and CamemBERT (a concurrent French BERT). Some baseline rows are taken from earlier papers and are marked as such in the source.

Text classification, CLS accuracy (%) tab. 3

Paraphrase, PAWS-X accuracy (%) tab. 4

Natural language inference, XNLI accuracy (%) tab. 5

Constituency parsing and POS tagging (F1, POS accuracy) tab. 6

Dependency parsing (UAS, LAS) tab. 7

Verb sense disambiguation, F1 (%) tab. 8

Noun sense disambiguation, F1 (%) tab. 9

Paper vs code

The repository is getalp/Flaubert (linked from the paper's footnotes) at commit ab48ced2aba5ba0933378fef8c9967a15d780ec1. The repository LICENSE is CC BY-NC 4.0, so the code is non-commercial; nothing from it is quoted here.

• Classification head. The paper describes dropout, linear, tanh, dropout, linear §5.1. The FLUE runner executes only dropout and one linear layer; the two-layer tanh version is commented out flue.py flue.py. The Hugging Face variant of the CLS config was not checked.
• CLS batch size. The paper states batch 16 §5.1. The example config for books uses batch 8 with 30 epochs and learning rate 5e-6 cls_books_lr5e6_xlm_base_cased.cfg. The example is one setting, not necessarily the one behind the reported numbers.
• Learning-rate decay. The paper states warmup and peak only §3.2. The repo uses an inverse-square-root decay after warmup optim.py README.md.
• Weight decay. The paper states 0.01 §3.2. The repo applies it as decoupled decay, scaled by the learning rate optim.py. The earlier gradient-based variant is commented out optim.py.
• Layer dropping. The paper says the authors "randomly drop a number of (attention) layers" §3.2. The active code drops the whole block: it runs the layer and then, with probability layerdrop, replaces its output with its input transformer.py. The per-layer early-skip version is commented out transformer.py. Compute is not saved in the active version.
• Pre-norm. The paper uses pre-norm for large only §3.2. Repo default is post-norm train.py; the post-norm branch is at transformer.py and pre-norm at transformer.py.
• Released base command. The repo gives one pre-training command, for base README.md. It sets dropout, GELU, batch and optimizer, but not pre-norm or layer dropping. The large-model command is not in the repository, so the large settings in the paper cannot be checked against code.
• Settings not in the paper. Dropout 0.1, GELU, maximum sequence length 512, masking rate 0.15 and the 80/10/10 split appear only in the repo command or defaults README.md train.py.
• Noun WSD ensemble. The paper says eight models per experiment, averaged §5.6. The training script trains one model per run 3.train_model.sh. The averaging step is in other scripts, which were not checked.
• Parsers. The dependency-parsing model §5.5 and the constituency parser §5.4 are not in this repository. The repository links external parser code only in its README.
• Data pipeline. The repo's cleaning and tokenization scripts exist preprocess.sh. The filter thresholds and the BPE vocabulary are not in the repo and were not verified against §3.1.

Pitfalls

• The head you implement changes the CLS numbers. The paper's head has an extra tanh layer, but the shipped runner does not. Pick one and say which in your reproduction.
• Large-model settings are paper-only. The repo has no large command, so warmup, learning rate, pre-norm and layer dropping for large cannot be checked against code.
• The layer-dropping sentence is ambiguous. "In all the experiments" could mean base too. The base config in the repo leaves it at 0.
• Weight decay is decoupled in the code. An implementation that adds weight decay to the gradient will not match the repo's behaviour.
• Inverse-square-root decay is unstated in the paper. Without it, post-warmup learning rates will differ a lot.
• Compute is large. The derived GPU-hour totals (about 13,120 base, about 49,920 large) are not in the paper; plan budgets accordingly.
• Verb WSD depends on the sense inventory. It has no trained parameters; its F1 depends on the Wiktionary examples and on cosine similarity over the model's contextual embeddings §5.6.
• Noun WSD changed the sense labels. Senses without a WordNet mapping were removed, and the training data is machine-translated §4.5. Results are not comparable to other WSD sets without that caveat.
• Common Crawl was heavily cleaned. The cleaning rules are described only as aggressive §3.1. Expect a corpus that differs from raw Common Crawl.
• Licensing is not settled for commercial use. The paper is CC BY 4.0, but the code is CC BY-NC 4.0 and the model weights are not covered by any license in the paper. The corpora come from WMT, OPUS and Wikimedia, each with its own terms §3.1. Check the weights' license and each corpus before commercial use.

Lineage

• Builds on: BERT's architecture and MLM objective §3.2; the RoBERTa training recipe, minus next-sentence prediction §3.2; the XLM code base §3.2 README.md.
• Compared against: mBERT and XLM-R on XNLI §5 tab. 5; CamemBERT, a French BERT published concurrently §2.2 §5.
• Superseded by: not stated in this paper.