Spanish Pre-trained BERT Model and Evaluation Data

In plain words

Many models that read text were trained on many languages at once, and Spanish had few ready-made resources of its own for building or testing language models. This paper trains a BERT-style model on Spanish text alone. BERT learns by hiding some words in a sentence and guessing them from the words around them. Here the hidden units are whole words, even when a word is split into several pieces. The authors also gathered a set of Spanish test tasks into one collection, in the spirit of the English GLUE benchmark. On most of those tasks the Spanish-only model did better than the multilingual alternative, but not on every task. These are the authors' own numbers.

TL;DR

BETO is a BERT-Base-sized encoder pre-trained only on Spanish text, in cased and uncased versions §3. The paper also compiles Spanish tasks into GLUES §4. Author-reported, it beats the best multilingual BERT on most tasks and sets new state of the art on POS and MLDoc, but trails it on some QA settings [§5.2, tab. 1, tab. 2]. The paper gives no code for training, so the training recipe can only be taken from the paper. Check the training-data licences before commercial use (see Pitfalls).

Problem

Finding resources to train or evaluate Spanish language models was hard §1. The usual choice was multilingual BERT, pre-trained on more than 100 languages at once §1. Single-language BERT models existed for several other languages and, the paper reports, often beat multilingual models in their own language §2. No Spanish BERT model had been released before this work §1, §2.

The approach assumes that a Spanish-only model is the better choice for Spanish. The limit is that a Spanish-only model cannot use English or other-language training data, which multilingual models can use §5.2.

Method

BETO is a Transformer encoder with the BERT-Base shape, pre-trained with masked language modeling on Spanish text only §3. It uses whole-word masking and dynamic masking, both borrowed from the BERT and RoBERTa line of work §3. A cased and an uncased model are trained §3. Fine-tuning follows the original BERT setup, with a sliding window for word-level tasks §5.1. The second contribution, GLUES, is a set of Spanish NLP tasks with their metrics §4.

1. Architecture. 12 self-attention layers, 12 heads each, hidden size 768, 110M parameters, similar in size to BERT-Base §3.
2. Corpus. All Spanish Wikipedia and all Spanish sources of the OPUS project, including UN and government journals, TED talks, subtitles and news. About 3 billion words, with a size comparable to BERT's original corpus. The authors released it §3.
3. Vocabulary. Byte-pair encoding through SentencePiece: 31K subwords plus 1K placeholder tokens for later use, giving 32K tokens, shared by both versions §3.
4. Masking. Dynamic masking: each sentence gets 10 different masks §3. Whole-word masking: when one sub-word of a word is masked, all contiguous sub-words of that word are masked too §3. The masking rate is not stated.
5. Pre-training schedule. 2M steps per model; peak learning rate 0.0001 with the first 10,000 steps as warm-up §3. Phase 1: the first 900k steps with batch size 2048 and maximum length 128. Phase 2: the remaining 1.1M steps with batch size 256 and maximum length 512 §3. Hardware: Google's preemptible TPU v3-8 §3.
6. Fine-tuning. Task-specific output layers, following BERT §5.1. No preprocessing except WordPiece tokenization with the learned vocabulary §5.1. Adam with β1 = 0.9 and β2 = 0.999, L2 weight decay 0.01, warm-up over the first 10% of steps, then linear decay §5.1. Maximum sentence length 128 tokens, to fit on a single GPU §5.1.
7. Word-level tasks. A sliding window: after the first window, the last 64 tokens are kept and only 64 new tokens are fed in §5.1.
8. Hyperparameter grid. Batch size {16, 32}, learning rate {5e-5, 3e-5, 2e-5}, epochs {2, 3, 4}, following BERT's recommendations. The paper leaves an extensive search to future work §5.1.
9. GLUES tasks §4:
   • XNLI: 3-way natural language inference. Training uses the Spanish machine translation of MNLI, with the XNLI dev and test sets. Metric: accuracy.
   • PAWS-X: paraphrase identification, Spanish portion. Metric: accuracy.
   • CoNLL-2002 NER: person, organisation, location and miscellaneous entities, with BIO tags. Metric: F1.
   • Universal Dependencies v1.4 POS: Spanish subset. Metric: F1.
   • MLDoc: four classes (CCAT, ECAT, GCAT, MCAT), largest Spanish training split. Metric: accuracy.
   • Universal Dependencies v2.2 dependency parsing: AnCora and GSD Spanish concatenated. Metrics: UAS and LAS. No results are reported in the paper.
   • Question answering: MLQA and TAR training sets, tested on MLQA and XQuAD. Metrics: F1 and exact match (EM) §4.

The sliding window and the masking follow the recipe below. The paper has no algorithm block, so this pseudocode is a reading of the prose, with the unstated settings marked.

# Whole-word masking of one tokenized sentence  [§3]
# Input:  tokens (sub-word list), word_of (word index of each sub-word)
# Output: masked tokens, and the positions the model must predict

def whole_word_mask(tokens, word_of, mask_rate):
    # The masking rate is not stated in the paper  [§3]
    # Choose whole words, not single sub-words  [§3]
    chosen_words = sample_words(word_of, mask_rate)
    # Mask every sub-word of a chosen word, not only the picked one  [§3]
    positions = [
        index
        for index in range(len(tokens))
        if word_of[index] in chosen_words
    ]
    masked_tokens = replace_with_mask(tokens, positions)
    # Whether masked positions use [MASK], a random token or the original
    # token is not stated  [§3]
    return masked_tokens, positions

# Two-phase pre-training with dynamic masking  [§3]
# Total length: 2M steps  [§3]

def pretrain(model, corpus):
    for step in range(2_000_000):
        # Phase 1: batch 2048, length 128, first 900k steps  [§3]
        # Phase 2: batch 256, length 512, the remaining steps  [§3]
        if step < 900_000:
            batch_size, max_length = 2048, 128
        else:
            batch_size, max_length = 256, 512
        # Whether batch size counts sequences or sentences is not stated  [§3]
        batch = sample_batch(corpus, batch_size, max_length)
        # Peak learning rate 1e-4; warm-up over the first 10,000 steps  [§3]
        # Shape of the rate after warm-up is not stated  [§3]
        learning_rate = schedule(step, peak=1e-4, warmup=10_000)
        loss = masked_lm_loss(model, batch)
        update(model, loss, learning_rate)

# Sliding window over one long word-level input  [§5.1]
# First window: 128 tokens; later windows add 64 new tokens  [§5.1]

def sliding_windows(tokens):
    for start in range(0, len(tokens), 64):
        # The last 64 tokens of a window open the next one  [§5.1]
        window = tokens[start : start + 128]
        yield window
        # Whether special tokens count toward 128, and how the final
        # window is handled, is not stated  [§5.1]

# Fine-tuning grid from BERT's recommendations  [§5.1]
# The rule for picking one configuration per task is not stated  [§5.1]

grid = [
    (batch_size, learning_rate, epochs)
    for batch_size in [16, 32]
    for learning_rate in [5e-5, 3e-5, 2e-5]
    for epochs in [2, 3, 4]
]

No equations, algorithms or figures appear in the compiled paper.

Hyperparameters

Results

Author-reported. Setup: BETO-cased and BETO-uncased, fine-tuned on each task's Spanish training data only tab. 1 caption. The baseline is the best multilingual BERT result the authors found in the literature for the same setting [tab. 1, tab. 2]. Whether the scores are dev or test is not stated §5.2.

Table 1 (accuracy, except NER and POS, which are F1) tab. 1:

Superscripts in the source: [a] wu2019beto (arXiv:1904.09077); [b] yang2019pawsx (arXiv:1908.11828). The paper marks new state of the art with an asterisk.

Table 2 (F1 / exact match) tab. 2:

Superscripts in the source: [c] lewis2019mlqa (arXiv:1910.07475); [d] carrino2019automatic (arXiv:1912.05200).

Headline reading, per the paper: BETO beats the best mBERT on most GLUES tasks. The exceptions are some QA settings §5.2, tab. 2. The paper highlights POS and MLDoc as new state of the art §5.2. Two results are cited in the text but not tabled: XLM-RoBERTa (560M parameters) reports 85.6% XNLI and 89% NER, using multilingual training data §5.2, arXiv:1911.02116. mBERT fine-tuned with the English PAWS-X training set reports 90.7% on the Spanish test set §5.2.

Paper vs code

The official code is the repository the paper links in its footnote §1 fn.. The owner name (dccuchile) matches the authors' Universidad de Chile affiliation [author block], and the README points to checkpoints hosted under the same group. The repository at code_commit contains only a README, a model download page, a licence, and two replication stubs. There is no pre-training or fine-tuning code.

• No training code. The replication stubs say only "This is work in progress" ner-c.md mldoc.md. Pre-training and fine-tuning settings can be taken from the paper only.
• Training length. The paper says 2M steps §3. The README also says 2M steps README.md. The model page lists checkpoints at 1.3M steps and at 2M steps, for both cased and uncased models.md. The paper does not describe the 1.3M-step checkpoints.
• Vocabulary size. The paper gives 31K subwords plus 1K placeholders, 32K in total §3. The README says "about 31k BPE subwords" README.md. The released vocab.txt is not in the repository, so its size could not be checked.
• Whole-word masking. The README confirms the technique README.md. It does not mention dynamic masking.
• Baseline for PAWS-X. Table 1 gives 89.00 for mBERT tab. 1 (b). The README lists 90.70 as the best multilingual result README.md. That figure is the general-training result that the paper gives in the text §5.2.
• Result values. The README's BETO numbers for POS, NER-C, MLDoc, PAWS-X and XNLI match Table 1 README.md.
• Dev or test. The README links the NER and MLDoc results as dev_results files README.md. The paper does not say whether Table 1 uses dev or test scores §5.2. Treat the numbers as unconfirmed on this point.
• Licence. The repository's LICENSE is CC BY 4.0 LICENSE. The README warns that the authors are not sure all training datasets have licences compatible with CC BY 4.0, "especially for commercial use" README.md.
• Other repositories. The corpus §3 fn. and GLUES §4 fn. are in separate repositories. They were not fetched, so nothing from them is used here.

Pitfalls

• Training data licences. The authors say the licences of the training datasets are unclear, especially for commercial use README.md. The corpus is drawn from OPUS sources, including subtitles and news §3. This card does not republish the model or data, but anyone using the weights or the corpus commercially must check each source. wikillm's own rule is "when in doubt, leave it out."
• Unstated pre-training details. Not stated: the masking rate, the replacement rule, the optimizer and its decay after warm-up, batch-size units, how sentences are packed into sequences, dropout, weight decay during pre-training, and the feed-forward size. An implementation has to guess these, and the choice affects the result.
• Phase switch. The paper says the first 900k steps use a different batch size and length than the rest §3. It attributes this schedule to a large-batch optimisation work by You et al. §3, but does not name the optimizer it used.
• No fine-tuning selection rule. The grid is given but not the chosen values per task. The paper does not say whether they were picked on dev data, nor how many seeds were run §5.1. Single-run gaps are small in places: uncased MLDoc is 0.42 points above the best mBERT, and uncased PAWS-X is 0.55 points above tab. 1. No variance is reported.
• Model choice depends on the task. Cased is best on XNLI, NER and POS, while uncased is best on PAWS-X and MLDoc tab. 1. Choose the variant per task, not once for the whole model.
• QA is weaker in some settings. On TAR-to-XQuAD, the best mBERT leads on F1 (77.60 vs 77.56) and exact match (61.80 vs 57.06) tab. 2. The paper says nearly half of the 81K MLQA examples have a mismatch between answer and start position §5.2. It suggests machine-translation quality as a likely cause, but does not say the mismatch was corrected.
• Dependency parsing is described but not evaluated. Section 4 describes the task and its metrics §4, but the compiled paper reports no parsing results.
• Source material not in the compiled paper. The .tex file has draft tables and a note after \end{document}, including "(running)" entries and XLM-RoBERTa comparisons. These are not part of the paper and are not used here.
• Context window. Whether the 128-token window counts special tokens is not stated §5.1.

Lineage

• Builds on: BERT, arXiv:1810.04805, which gave the architecture, the masked language modeling objective and the fine-tuning recipe §1, §3, §5.1. RoBERTa, arXiv:1907.11692, which gave the dynamic masking and the larger batches §3. GLUE, the English benchmark that inspired GLUES §4.
• Compared against: multilingual BERT, with the best reported results from wu2019beto (arXiv:1904.09077), yang2019pawsx (arXiv:1908.11828), lewis2019mlqa (arXiv:1910.07475) and carrino2019automatic (arXiv:1912.05200) tab. 1, tab. 2. XLM-RoBERTa, arXiv:1911.02116, is compared in the text only §5.2.
• Superseded by: not stated in this paper. The authors say that Spanish ALBERT models were in progress §6.