ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

In plain words

Language models such as BERT learn about words by filling in blanks in large amounts of text. Most models that handle Arabic were trained mainly on formal, written Arabic. Arabic also has many spoken dialects, and these are under-studied because there is little data for them.

The paper trains two models of BERT's standard size. ARBERT learns from a broad collection of formal (Modern Standard) Arabic. MARBERT learns from a large set of Arabic tweets, which mix formal Arabic with dialects. The authors also assemble ARLUE, a benchmark of 42 datasets grouped into six task clusters, so that many Arabic understanding tasks are tested in one standard setup.

On that benchmark, the tweet-trained model MARBERT-v2 gets the highest overall score, ahead of a much larger multilingual model. These numbers come from the authors' own experiments.

TL;DR

• Two BERT-Base-sized Arabic encoders: ARBERT (formal Arabic, 61 GB) and MARBERT (1B tweets) §3.1 §3.2.
• Both use a 100K WordPiece vocabulary. MARBERT drops the next-sentence prediction objective §3.2.
• ARLUE covers 42 datasets in six task clusters. MARBERT-v2 scores 77.40 and XLM-R Large 76.55, author-reported §5.2.
• Read the pitfalls first: the 77.40 score averages eight table rows, "MARBERT-v2" is never defined, and the checkpoints are for research use only.
• Use it as an Arabic encoder baseline. MARBERT's pre-training data is not released, so MARBERT cannot be rebuilt from this card.

Problem

Multilingual masked language models cover Arabic as one of many languages. Their Arabic share is small: mBERT has 5K Arabic WordPieces out of 110K, and XLM-R has 14K out of 250K §3.1. The abstract also names two costs of these models: they are expensive at inference time, and their pre-training data for non-English languages is limited in size and diversity [abstract].

AraBERT is trained on MSA only, and mBERT and XLM-R see mostly MSA §3.2 §3.3. The authors state that dialects are under-studied because resources are rare, so these models are not well suited to dialectal downstream tasks §3.2. To address this, they pre-train MARBERT on Arabic tweets §3.2.

The authors also want standardized experiments across many Arabic varieties and tasks, under rich conditions §5.

Method

The paper trains two models with the same BERT-Base encoder and differs only in data and objective. Both are then fine-tuned on each dataset and on each ARLUE task cluster §3.4 §5.2.

1. Pre-processing (both models). Remove Arabic diacritics. Replace every URL, user mention and hashtag with the literal tokens URL, USER and HASHTAG. No other normalisation is applied, for example of letter variants §3.1 §3.2.
2. Tokenisation (both models). Use the BERT WordPiece tokenizer with a 100K-entry vocabulary §3.1 §3.2. The paper does not state how the vocabulary was built.
3. Masked-LM objective (ARBERT). Follow BERT's pre-training setup with whole-word masking. Select 15% of the N input tokens. Replace 80% of the selected tokens with [MASK], 10% with a random token, and keep 10% unchanged §3.1.
4. Masked-LM objective (MARBERT). Same architecture as BERT-Base, without the next-sentence-prediction (NSP) objective, because tweets are short §3.2. The masking details are not restated for MARBERT.
5. Training data, ARBERT. About 61 GB of MSA text, roughly 6.5B tokens, from the seven sources in tab. 1. The table lists 32 MB of Egyptian Arabic (OSCAR-Egyptian) among them §3.1 tab. 1.
6. Training data, MARBERT. Randomly sample 1B Arabic tweets from an in-house set of about 6B tweets. Keep a tweet if it has at least three Arabic words, judged by character string matching. Non-Arabic text is not removed. The sample is 128 GB, or 15.6B tokens §3.2.
7. Fine-tuning (both models). Adam, sequence length 256, batch 32, learning rate 2e-6, 25 epochs. Pick the best epoch on the development set §3.4. Pseudocode below.
8. ARLUE. Concatenate the train, dev and test splits of each cluster. The social meaning datasets stay separate. The ARLUE score is the macro-average over the table rows (see Results) §5.1 §5.2.

Pre-training objective, as written in the paper:

# Build one pre-training example  [§3.1] [§3.2]
# 15% of the N input tokens are selected; whole words are masked  [§3.1]
def make_masked_example(tokens, vocabulary):
    selected = pick_15_percent_of_words(tokens)
    inputs = list(tokens)
    for position in selected:
        # 80% [MASK], 10% a random token, 10% kept unchanged  [§3.1]
        roll = uniform(0.0, 1.0)
        if roll < 0.8:
            inputs[position] = MASK_TOKEN
        elif roll < 0.9:
            inputs[position] = random_token(vocabulary)
    targets = {p: tokens[p] for p in selected}
    return inputs, targets

Fine-tuning, as written in the paper:

# Fine-tune one dataset or ARLUE cluster  [§3.4]
# Adam; sequence length 256; batch 32; learning rate 2e-6; 25 epochs  [§3.4]
# NER: length 128, batch 16, lr 2e-6. QA: length 384, batch 16, lr 3e-5  [§3.4]
def finetune(model, train, dev):
    best_score = None
    for epoch in range(25):
        for batch in batches(train, size=32):
            # Loss function not stated in the paper
            loss = task_loss(model(batch.inputs), batch.labels)
            adam_step(model, loss, learning_rate=2e-6)
        # Select the epoch on dev data; for ARLUE, dev only  [§3.4]
        score = evaluate(model, dev)
        best_score = keep_best(best_score, score)
    return best_score

Hyperparameters

Results

Author-reported. Setup: fine-tuned on the TEST splits of ARLUE, one score per task cluster, compared with mBERT, XLM-R Base and Large, and AraBERT §5.2. The table below is copied from §5.2. Each cell is Acc / F1, except QA (EM / F1). Per-dataset results and the DEV tables are in §4 and the appendix.

Source: §5.2. The Senti row uses Acc / F1^PN. The SM row is the average over the social meaning tasks §5.2. The paper's text names MARBERT-v2 as the model with the highest score, followed by XLM-R Large (76.55) and ARBERT (76.07) §5.2.

The abstract claims new state of the art on 37 of 48 classification tasks across 42 datasets. It also states that XLM-R Large is about 3.4 times larger than the authors' models [abstract] §3.4. The per-task breakdown is in §4.

Paper vs code

No training or fine-tuning code is in the repository at commit 0d25dfe7cb2c48ebd11755351c708e539f2676b3. It holds a README, an image and two example TSV files for AJGT sentiment README.md. The checkpoints are on Hugging Face under UBC-NLP/ARBERT and UBC-NLP/MARBERT README.md. The ARLUE data is not in the repository at this commit.

The README and the paper disagree on several figures:

• ARBERT pre-training tokens: README says 6.2B README.md README.md; the paper text says 6.5B §3.1. Table 2 says 6.2B tab. 2.
• Benchmark size: README says 41 datasets, 5 tasks and 37 of 45 classification tasks README.md README.md. The paper says 42 datasets, six task clusters and 48 classification tasks [abstract] §5.
• MARBERT parameters: README says MARBERT uses ARBERT's architecture, which it gives as about 163M README.md README.md. The paper text says about 160M for MARBERT §3.2.

The README's per-dataset test tables were not checked against the paper's tables.

Pitfalls

• ARLUE score weighting. The score is not a six-cluster macro-average despite the abstract's wording. The table has eight rows: Dialect is split into three (binary, region, country). Averaging each row's two metrics and then the eight rows gives 77.40 for MARBERT-v2, which matches the reported score. Weighting the dialect rows as one cluster would give about 78.6 (our calculation from §5.2). Use the table rows, not the abstract's cluster count, when you compare numbers.
• MARBERT-v2 is undefined. The results table and the 77.40 score refer to MARBERT-v2. The paper body never says how it differs from MARBERT (data, steps, or objective). The conclusion says it was pre-trained on dialectal Arabic §7. Treat "MARBERT-v2" as a label only.
• Checkpoints are research-only. The README says the Hugging Face checkpoints are for research use, and that commercial use requires contacting the authors README.md. Do not ship or serve them in a commercial product on the strength of this card.
• MARBERT data cannot be rebuilt. The 6B-tweet collection is in-house. The ethics section says the pre-training datasets will not be released §3.2 [Ethical Considerations].
• Token and parameter counts conflict. ARBERT is 6.5B tokens in the text and 6.2B in the table and README §3.1 tab. 2 README.md. MARBERT is about 160M parameters in the text and 163M in the table §3.2 tab. 2.
• Table 1 sentence counts do not add up. The per-source sentence counts sum to about 305M, but the total row says 203.5M tab. 1.
• ARBERT is not purely MSA. The paper describes ARBERT as MSA, but Table 1 includes 32 MB of Egyptian Arabic (OSCAR-Egyptian) §3.1 tab. 1. The table lists seven rows, while Table 2 says "6 sources" tab. 2.
• Epoch counts do not match the step counts. At 256 sequences of 128 tokens, 8M steps is about 262B tokens. Over 6.5B tokens, that is about 40 epochs, not the 42 the paper states §3.1. The MARBERT figures (17M steps, about 36 epochs) check out §3.2.
• Whether ARBERT uses NSP is not stated. The paper says it follows BERT's pre-training setup §3.1, and only MARBERT is said to drop NSP §3.2.
• Model selection on test. For datasets with no dev split, the best epoch was chosen on the test set. For ARLUE, the best epoch was chosen on dev only §3.4 footnote. Scores on those datasets are optimistic.
• Learning rates were chosen by hand. The fine-tuning values were chosen in initial experiments on the dev data of a few tasks §3.4. The paper does not report a sweep per task.
• Single runs and no variance. The paper reports no seeds, repeats or confidence intervals for any table §5.2.
• QA is weak for MARBERT. The authors attribute MARBERT's QA result to its short input length §5.2. Fine-tuning uses length 384 for QA §3.4. Expect the same weakness if you use MARBERT for QA.
• Unstated details. Loss functions, optimizers after pre-training, warmup, weight decay, dropout, NER label decoding and the QA output head are not stated. An implementer must check the BERT TensorFlow code or the released checkpoints.
• Bias and privacy. The authors say the models may carry untested biases and that Twitter data is subject to platform policy [Ethical Considerations] README.md.

Lineage

• Builds on: BERT (Devlin et al., 2019), for the architecture, WordPiece tokenisation and masked-LM objective. This paper drops NSP for MARBERT §3.2.
• Compared against: mBERT, XLM-R (Base and Large) and AraBERT §3.3.
• Superseded by: not known from wiki content.