AraT5: Text-to-Text Transformers for Arabic Language Generation

In plain words

A large multilingual model such as mT5 covers Arabic only as one of more than 100 languages. Arabic is spoken in many regional varieties, and much of what people write online is informal and mixed with other languages. This paper asks whether a model trained on Arabic text does better than that multilingual model, which saw much less Arabic.

The authors train three small "fill in the blanks" models of the T5 family. Parts of each sentence are hidden, and the model learns to write the missing pieces back. One version learns from formal news and books, one from social media posts, and one from both. The same model is then retrained on a set of Arabic writing tasks, such as translation, summarising, writing headlines, and paraphrasing. The authors also collect these tasks into one benchmark, so that Arabic generation can be measured in a consistent way.

The models come out ahead of the multilingual baseline on most of the test sets, even though they saw less Arabic text. The paper matters to anyone who builds Arabic text generation and needs a starting point, or a yardstick.

TL;DR

AraT5 is a family of three Arabic T5-base encoder-decoder models, pre-trained on MSA text, Arabic tweets, or both. They are compared with mT5 on ARGEN, a new seven-task Arabic generation benchmark, and on the ARLUE understanding benchmark. The authors report that the models beat mT5 on 52 of 59 ARGEN test sets while using about 49% less Arabic pre-training data. Results are author-reported. Check Pitfalls before using any single number, because several text claims do not match the tables.

Problem

mT5 covers Arabic only as one of more than 100 languages, and the paper argues that its Arabic share and data quality are limiting §1 §5.1. Arabic has many dialects and much informal, code-switched text. The authors know of no earlier pre-trained sequence-to-sequence model for Arabic §1. There was also no benchmark for measuring Arabic generation across tasks §1. The paper's own mC4 check finds 13.59% non-Arabic paragraphs in a sample of Arabic mC4, mostly boilerplate or foreign text, and almost no real code-switching app. A. The authors' Twitter data, in contrast, is 28.39% dialectal in their sample §2.1.

Method

Models. Three T5-base encoder-decoder models, each with 12 layers per stack, 12 attention heads and 768 hidden units, about 220M parameters §2.3 fig. 1. The feed-forward inner size is 3072 and the key/value size is 64 (footnote in §2.3). The three variants are:

• AraT5_MSA: trained on MSA text only §2.3.
• AraT5_Tw: trained on Arabic tweets only §2.3.
• AraT5: trained on MSA and tweets together §2.3.

Pre-training data §2.1. MSA: 70 GB, 7.1B tokens, from AraNews, El-Khair, Gigaword, OSCAR, OSIAN, Arabic Wikipedia and Hindawi Books §2.1. Twitter: 1.5B tweets sampled from about 10B, kept if they have at least three Arabic words, 178 GB and 21.9B tokens. Together, 29B tokens, against 57B Arabic tokens for mT5 §2.1. About 4.14% of 1M sampled tweets contain non-Arabic strings §2.1.

Pre-processing §2.2. Remove diacritics. Replace URLs with <URL> and user mentions with <USER>. Strip HTML tags, elongation and hash signs. Reduce repeated characters, emojis and emoticons to one. The paper does not say how the reduction is done.

Vocabulary §2.2. A SentencePiece vocabulary of 110K pieces, built from 70M MSA sentences, 200M Arabic tweets, 15M English Wikipedia sentences and 5M sentences from ten other Wikipedias. It therefore covers 11 non-Arabic languages §1. The text calls the pieces "WordPiece tokens" §2.2, which conflicts with the SentencePiece name. See Pitfalls.

Pre-training objective §2.3. T5 span corruption. About 15% of input tokens are dropped at random, and each run of consecutive dropped tokens is replaced by a single sentinel token. The model learns to reconstruct the original sequence. The paper does not give the mean span length.

# Span corruption for one pre-training sentence  [§2.3]
# Input:  tokens: list of subword ids for one sentence
# Output: corrupted: model input with sentinels
#         target: the original sequence the model must rebuild

def corrupt_sentence(tokens):
    # Drop about 15% of positions at random  [§2.3]
    dropped = sample_positions(len(tokens), rate=0.15)
    # Replace each run of consecutive dropped positions
    # by one sentinel token  [§2.3]
    corrupted = replace_runs_with_sentinels(tokens, dropped)
    # The model is trained to rebuild the original tokens  [§2.3]
    target = tokens
    return corrupted, target

Fine-tuning and selection §4. Each ARGEN task is fine-tuned on its training split. mT5 is fine-tuned on the same data as a baseline. The best checkpoint is chosen on the dev split, then blind-tested once on the test split. The paper does not describe the fine-tuning optimiser or schedule. The repository code and logs do (see Hyperparameters).

# Fine-tune one ARGEN task, keep the best dev checkpoint, test once  [§4]
# Input:  pretrained model; train, dev, test splits; task metric
# Output: test score of the dev-selected checkpoint

def finetune_and_test(model, train, dev, test, metric):
    # Fine-tune on the task training split  [§4]
    checkpoints = train_seq2seq(model, train)
    # Pick the checkpoint with the best dev score  [§4]
    best = max(checkpoints, key=lambda c: metric(c, dev))
    # Blind-test the chosen checkpoint once  [§4]
    return metric(best, test)

ARGEN benchmark §3. Seven tasks from 19 datasets, with 59 test splits. Nine of the datasets are new §1. The tasks are:

• Machine translation: 46 test splits. 42 are Arabic to English, from UN, IWSLT and AraBench (ADPT, MADAR, QAraC, Bible) §3.1 §4.1. Four are X to Arabic (English, French, German, Russian), built from OPUS with 1M training pairs each §3.1.
• Code-switched translation: six test splits. Two natural sets (Algerian Arabic-French and Jordanian Arabic-English, 300 tweets each, manually translated). Two synthetic sets (MSA-EN and MSA-FR), each in two directions, made from UN test data by translating about 30% of the Arabic n-grams with mBART §3.2.
• Text summarisation: EASC (153 articles, 5 extractive summaries each) and the Arabic part of WikiLingua §3.3.
• News title generation: 120K AraNews articles with titles of at least three words, split 80/10/10 §3.4.
• Question generation: 96K (passage, answer, question) triples from ARCD, XTREME, MLQA, XQuAD and TyDi QA §3.5.
• Paraphrasing: AraPara, a new 122K-pair set built from OPUS, with 116K train and 6K dev. Test sets are ASEP (603 pairs) and APB (1,010 pairs) §3.6 app. C.
• Transliteration: BOLT Egyptian Arabic SMS and chat. The source is romanised Arabizi or Egyptian Arabic, and the target is the Egyptian transliteration. Split: 58.9K train, 5.4K dev, 5.4K test §3.7.

ARLUE. The understanding benchmark ARLUE is fine-tuned in single-task and multi-task settings §4.6 app. B.

Hyperparameters

Pre-training values come from the paper. The fine-tuning rows marked "code" or "notebook" are not in the paper: they come from the repository, and they are the authors' example settings, not the settings used for the paper's tables.

Results

Author-reported. Scores are on the test split unless noted. BLEU for generation tasks, ROUGE for summarisation, ARLUE score for understanding. "Best AraT5" is the highest of the three AraT5 variants in the cited table.

The paper reports that the models beat mT5 on 52 of 59 ARGEN test sets §7. mT5 is higher on several sets, including EASC and the JOR-EN code-switched set. Multi-task ARLUE training beats single-task training on 7 of 9 dialect experiments and 18 of 36 social-meaning experiments app. B.

Paper vs code

The repository (UBC-NLP/araT5, commit 17ec307) contains only the fine-tuning script, a notebook, two sample title-generation files, and a README. It has no pre-training code, no vocabulary, no full ARGEN data, and no evaluation script for the paper's tables. Each item below compares what the paper says with what the repository does.

• Pre-training not released. The 1M-step run, the optimiser and the vocabulary build are absent from the repo. Pre-training values come from the paper only §2.3. The README links released checkpoints on Hugging Face [README.md#L249-L257].
• ARGEN not released in the repo. The paper says the datasets are accessible from the repository §7. The repo holds only sample title-generation data, about 1,000 training and 100 validation lines ARGEn_title_genration_sample_train.tsv ARGEn_title_genration_sample_valid.tsv.
• Fine-tuning values. The paper does not state them. The README example uses learning rate 5e-5, batch 8, lengths 128 and 3 epochs [README.md#L164-L169]. The notebook logs the same learning rate and a linear schedule Fine_tuning_AraT5.ipynb. The script's default source length is 1024 run_trainier_seq2seq_huggingface.py.
• Model selection. The paper picks the best dev checkpoint §4. The script loads the best model by eval_bleu [README.md#L173] and stops after 20 epochs without improvement, a value hard-coded in the script run_trainier_seq2seq_huggingface.py run_trainier_seq2seq_huggingface.py.
• BLEU and ROUGE. The paper says BLEU and ROUGE without a variant. The script uses sacreBLEU corpus BLEU run_trainier_seq2seq_huggingface.py. ROUGE uses the rouge_score package with stemming, scaled by 100 run_trainier_seq2seq_huggingface.py. Sentences are split for ROUGE only run_trainier_seq2seq_huggingface.py.
• Decoding. Beam size is not stated in the paper. The script defaults to no beam setting run_trainier_seq2seq_huggingface.py. The README's inference example uses sampling with top-k 120 and top-p 0.95 [README.md#L191-L199]. That is an example, not the paper's evaluation.
• Label masking. Padded label positions are set to -100 and excluded from the loss by default run_trainier_seq2seq_huggingface.py run_trainier_seq2seq_huggingface.py.
• Task prefix. The paper does not describe one. The script adds an optional source prefix that defaults to none run_trainier_seq2seq_huggingface.py run_trainier_seq2seq_huggingface.py.
• Model architecture not checkable. The repo has no model definition or pre-training code, so the architecture and pre-training values can be checked only against the paper.

Pitfalls

• Text and tables disagree in several places. Use the tables, and recheck any single claim before you cite it:
  • Section 4.4 says all models beat mT5 on NTG and QG. For QG, AraT5_Tw (12.06) and AraT5_MSA (14.18) are below mT5 (15.29) §4.4.
  • Section 4.5 says AraT5_MSA reaches 17.52 BLEU on APB. The table gives 19.43 for APB and 19.38 for ASEP §4.5.
  • Section 4.5 says the transliteration result beats the prior SOTA by 7.1 points. The table caption gives that SOTA as 65.88, the same as AraT5_MSA's result §4.5.
  • Section 4.1 gives BLEU gains of +5.25, +4.99 and +0.45 over S2S-2M, S2S-10M and mT5. The average row gives +5.25, +4.50 and +0.55 §4.1 tab. 2.
  • The caption of the Arabic-to-English table says "English to Arabic" tab. 2. Its rows and the text are Arabic to English.
  • The abstract and conclusion say the models are better on "all ARGEN tasks" with 52 of 59 test sets. mT5 is ahead on EASC overall §4.3. No significance test is reported, and "significantly" is not backed by one.
• Data table does not add up. The MSA rows sum to about 7.47B tokens and about 310M sentences. The stated totals are 7.1B and 205.4M §2.1. The OSCAR-Egyptian row lists 101.9M sentences for 3.8M tokens, which looks like a typo. The Twitter and combined sentence counts are still "xx" placeholders.
• Vocabulary method is unclear. The text says SentencePiece, then WordPiece tokens §2.2. The algorithm and its settings are not stated.
• Pre-training optimiser not stated. Only the learning rate is given. The optimiser, schedule and warmup are not stated; do not fill them in from the original T5 paper.
• CST size. The text says four code-switched sub-test sets §3.2. The results table has six test splits §4.2, and only six makes the stated total of 59 test splits add up.
• Evaluation not reproducible from the paper alone. The BLEU variant, tokenisation, ROUGE settings, beam size and fine-tuning hyperparameters are not stated. Use the code values only if you also state that they come from the code.
• "Zero-shot" means no pre-training on those languages. The X-to-Arabic models were still fine-tuned on 1M parallel pairs per language §4.1 §3.1.
• SOTA comparisons are not like for like. The SOTA systems in the tables were trained on much more parallel data, for example about 42M sentences §4.1. The authors fine-tuned on 2M.
• README does not match the paper. The README describes an ARGEN with four tasks and ten datasets, and an AraT5v2 with 1024-token inputs and 100 sentinels [README.md#L15-L26] [README.md#L70-L72]. Its code-switching table has rows shifted: the "MSA-FR to FR" row carries the paper's MSA-EN to EN values, and the "MSA-EN to EN" row carries the paraphrase II values [README.md#L146-L149]. Use the paper's tables and not the README's.
• Checkpoint use restricted. The README says the released checkpoints are for research only, and commercial use needs the authors' permission [README.md#L247]. This applies to the models, not to this card. The card stays CC BY 4.0. Check the Hugging Face model pages before using the weights.

Lineage

• Builds on: T5 (arXiv:1910.10683), the text-to-text framework and span-corruption objective used here §2.3.
• Compared against: mT5 (arXiv:2010.11934), fine-tuned on the same data as baseline II §4.1. Vanilla Transformers trained from scratch with Fairseq are baseline I §4.1.
• Compared against: ARLUE and MARBERT, the Arabic understanding benchmark and model, for the understanding results §4.6 app. B. The paper gives no arXiv id for them.
• Related follow-up named in the README: AraT5v2 with longer inputs and more data [README.md#L15-L26]. The paper does not describe it, so no numbers from it are used here.