mGPT: Few-Shot Learners Go Multilingual

In plain words

Most large language models are trained mainly on English text, and they work much less well in other languages. This paper builds a text generator, trained to predict the next word, that covers 61 languages from 25 language families. Many of these languages are low-resource, which means they have little training text. Several are spoken in the Commonwealth of Independent States or by small peoples in Russia.

The model is built like GPT-3, which means it can be used without any extra training. You give it a few worked examples or a plain task description in the prompt, and it continues the text. For a classification task, the answer is the label whose sentence the model finds most likely. The authors release two sizes, about 1.3 billion and 13 billion parameters.

The results are mixed. The smaller model holds up well against a similar multilingual model with fewer languages. The larger model is not clearly better, and in few-shot settings it often falls behind that comparison model.

TL;DR

• Decoder-only Transformers, mGPT-1.3B and mGPT-13B, follow the GPT-3 layout and are pretrained on Wikipedia and C4 in 61 languages §3.1 §3.3 tab. 1.
• The tokenizer is byte-level BPE with the "default" strategy, chosen by held-out perplexity over four variants §3.2 tab. 3.
• Author-reported: mGPT-1.3B is on par with or better than XGLM-1.7B on most classification tasks §5.3. mGPT-13B beats XGLM-7.5B zero-shot on four of five classification tasks, but trails it on most few-shot settings tab. 7.
• The main results are classification and NER/POS. mGPT scores are close to random on hate speech, and the NER and POS baselines are fine-tuned models, so they are not a like-for-like comparison §4.2 tab. 8 tab. 9 tab. 10.
• The compiled code covers evaluation and inference only. Pretraining code and several training details are not released, so the pretraining numbers cannot be checked against code [code tree] §3.4.

Problem

The authors argue that few-shot prompting of autoregressive models is well studied in English, but multilingual autoregressive models and low-resource languages are not §1. Fine-tuning needs labelled data, which is costly for new languages and tasks, and it is also expensive in compute §1. Massively multilingual models such as XLM-R, mBART and mT5 cover many languages §2. The paper claims to present one of the first multilingual autoregressive models covering more than 61 languages §2. XGLM is the closest prior multilingual GPT-style model, with 30 languages §2.

The paper asks whether a GPT-style model can cover more languages, including Commonwealth of Independent States and small Russian peoples' languages, and still give useful zero- and few-shot performance §1 §4.

Method

Overview. The paper trains two causal language models, mGPT-1.3B and mGPT-13B, on a multilingual corpus. It then evaluates them on language modelling, classification, sequence labelling, knowledge probing, external benchmarks and generation diversity §1 §4.

Pretraining data. Each step below follows §3.1.

1. Collect documents from the Wikipedia dump v. 20201101, extracted with WikiExtractor, and from C4 downloaded via TensorFlow Datasets.
2. Deduplicate: hash each text with a 64-bit hash and keep only texts with a unique hash.
3. Filter C4 by following the cited procedure for C4 filtering §3.1. Then discard the most strongly and the most weakly compressing deduplicated texts. The zlib compression ratio range for accepted text is ×1.2 to ×8 §3.1.
4. Train a binary quality classifier with Vowpal Wabbit. Wikipedia documents are positive examples and filtered C4 documents are negative examples §3.1.
5. Apply a set of language-agnostic cleaning heuristics. The paper does not list them §3.1.

The corpus is 46B (Wikipedia) and 442B UTF characters (C4), about 600 GB in total §3.1. Languages were chosen to narrow the gap between high- and low-resource languages: 20 languages come from the tail of the C4 language list, plus underrepresented languages of Russia and official or resource-lean CIS languages §3.1 tab. 1. The language list is in tab. 1. Its family counts add up to 61 languages in 25 families, and Mandarin is commented out in the source.

Tokenization. Five strategies were compared by training small models and measuring held-out perplexity §3.2. They differ in how uppercase letters, digits, arithmetic operators, punctuation and whitespace are represented §3.2 tab. 2:

• default: byte-level BPE §3.2.
• case: each uppercase character becomes a special case token followed by the lowercase character §3.2.
• arithmetic: case, plus numbers and arithmetic operators as single tokens §3.2.
• combined: arithmetic, plus punctuation and whitespace as single tokens §3.2.
• char: character-level tokens §3.2.

The default strategy had the lowest average perplexity tab. 3. The paper uses it for both final models §3.2.

The perplexity used for this comparison is defined in eq. 1. It is normalised by the number of characters, because tokenizers produce different token counts for the same text §3.2.

PPL(t) = \exp\left(-\frac{1}{|c|}\sum_{i=0}^{|t|}\log p_\theta(x_i \mid x_{<i})\right) eq. 1

Here t is the input text, |t| its length in tokens and |c| its length in characters §3.2.

Architecture. The architecture follows GPT-3. The authors use the GPT-3 description, the GPT-2 code base from Hugging Face and Megatron-LM §3.3. The model sizes follow the GPT-3 1.3B layout (24 layers, hidden size 2048) for mGPT-1.3B and the GPT-3 13B layout (40 layers, hidden size 5120) for mGPT-13B tab. 4. The layers alternate dense and sparse attention, as in GPT-3 §3.3 tab. 4.

Pretraining. The procedure mostly follows GPT-3 §3.4. The authors use DeepSpeed and Megatron-LM, a total batch size of 2048, a context window of 512 tokens and 600k training steps. The models saw 400B tokens in total §3.4. The 1.3B model trained for 14 days on 256 V100 GPUs, and the 13B model for 22 days on 512 V100 GPUs §3.4.

Classification. Each example is scored once per label. The label is the one whose completed prompt has the lowest summed negative log-likelihood over its tokens §4.2.1. Demonstrations for few-shot settings are drawn from the training data and placed before the example §4.2.1. The prompt templates are in tab. 6.

# Zero/few-shot classification by label likelihood  [§4.2.1]
# Input:  one example, label prompts (label -> verbalizer), demonstrations
# Output: predicted label

def classify(example, label_prompts, demonstrations, model):
    # One candidate per label: the demonstrations, then the example with
    # that label's verbalizer filled in  [§4.2.1] [tab. 6]
    candidates = []
    for label, verbalizer in label_prompts.items():
        text = demonstrations + fill_prompt(example, verbalizer)
        candidates.append(text)

    # Summed NLL over all tokens of each candidate  [§4.2.1]
    nll_per_label = []
    for text in candidates:
        token_losses = model_token_losses(text)
        nll_per_label.append(sum(token_losses))

    # The label with the lowest summed NLL is the prediction  [§4.2.1]
    best = argmin(nll_per_label)
    return list(label_prompts)[best]

Sequence labelling. For NER and POS, the model tags words one at a time, left to right §4.2.2. The prompt holds the earlier words with their tags, then the current word followed by a blank. Each possible tag fills the blank, and the tag with the lowest summed loss is kept §4.2.2 sequence_labeling_prediction.ipynb. The first word is used only as context §4.2.2. The paper ran zero-shot and 4-shot, but reports only the 4-shot results §4.2.2.

# Word-by-word tagging by label likelihood  [§4.2.2]
# Input:  words of one sentence, tag set, demonstrations (4-shot)
# Output: one tag per word

def tag_sentence(words, tag_set, demonstrations, model):
    # Tags decided so far, one per word already processed  [§4.2.2]
    decided_tags = []
    for position in range(len(words)):
        # Earlier words with their tags, then the current word and a blank
        # placeholder  [§4.2.2]
        prefix = render_prefix(words, decided_tags, position)
        # Fill the blank with each tag and score each full string by summed
        # NLL  [§4.2.2]
        candidates = [demonstrations + prefix + tag for tag in tag_set]
        nll_per_tag = [sum(model_token_losses(text)) for text in candidates]
        decided_tags.append(tag_set[argmin(nll_per_tag)])
    return decided_tags

Knowledge probing. The paper uses the mLAMA dataset in 23 languages §4.3. Each query is a knowledge triple turned into a cloze-style template. Each query has three candidate objects: the true one and two sampled from the source knowledge base. Accuracy is precision@1 averaged over relations, per language §4.3.

External and generation evaluation. The authors cite external evaluations on SuperGLUE tasks by the BLOOM authors and on a shared task on multilingual clause-level morphology §4.4 fig. 6. For generation, seven lexical diversity metrics are computed on 100 story-generation samples in five languages, with the sampling settings in §4.5.

Hyperparameters

Results

Author-reported. All numbers below come from the paper. The only results from another group are the external comparisons noted below.

Classification (zero- and few-shot, accuracy in %, averaged across languages). Setup: XWINO, PAWSX, XCOPA, XNLI and hate speech. The baselines XGLM-1.7B and XGLM-7.5B were reproduced by the authors with the released XGLM weights and fairseq, using the templates of Lin et al. §4.2.1 tab. 7. The ✗ cells are settings the authors marked as not reported by XGLM and not run here §4.2.1. The paper says that some XGLM scores were reproduced in zero-shot, while PAWSX and XNLI did not reproduce §4.2.1.

tab. 7 §4.2.1

The paper's own summary: mGPT-1.3B is comparable to XGLM-1.7B while having fewer weights and covering twice as many languages §4.2.1. mGPT-13B beats XGLM-7.5B zero-shot on all tasks except XNLI, and it beats XGLM-7.5B few-shot only on PAWSX and XNLI §4.2.1 tab. 7.

Hate speech by language (accuracy in %). Scores are close to the 50% chance level for binary labels, and the paper calls them close to random guessing §4.2.1. The 1-shot and 16-shot settings were not run tab. 8.

tab. 8 §4.2.1

NER (F1, 4-shot mGPT vs fine-tuned baselines). The baselines are fine-tuned on the training set, and the mGPT scores are 4-shot, so they are not directly comparable §4.2.2 tab. 9.

tab. 9 §4.2.2

POS-tagging (accuracy in %, 4-shot mGPT vs fine-tuned baselines). The first block covers the 18 XGLUE languages. The second block covers the seven CIS and low-resource UD languages, where the baselines were not available (✗) tab. 10 §4.2.2. The averages are over the XGLUE block only tab. 10.

tab. 10 §4.2.2

The authors say mGPT-1.3B outperforms mGPT-13B on practically every language, with Italian as the exception §4.2.2 tab. 10. On the CIS and low-resource languages, the 1.3B model stays close to its XGLUE level, while the 13B model is lower §4.2.2.

Language modelling (perplexity). The per-language and per-family perplexity results are only in figures fig. 3 fig. 4. The text says mGPT-13B reaches its best perplexities, in the 2-to-10 range, for most languages. Only seven languages have values up to about 20 §4.1 fig. 3. Family-level results show mGPT-13B with consistently lower perplexities than mGPT-1.3B. The 1.3B model is worse on the Basque, Greek, Kartvelian and Turkic families §4.1 fig. 4.

Tokenizer ablation (average perplexity over languages, lower is better). This is the only table that compares the tokenizer strategies tab. 3.

tab. 3 §3.2

Correlation tests on language modelling. The table reports p-values for three comparisons. It does not report correlation coefficients tab. 5 §4.1.

tab. 5 §4.1

Knowledge probing (precision@1, 23 languages). Random baseline is 0.33. The text says mGPT-1.3B and mGPT-13B score at least 0.6 on Germanic, Romance, Austro-Asiatic, Japonic and Chinese, and below 0.5 on Uralic, Slavic, Koreanic and Afro-Asiatic. Compared with XGLM-7.5B, the models are lower in almost all languages, and they match GPT-3 Curie 6.5B §4.3 fig. 5. The per-language values are only in the figure.

External evaluation (third-party, as reported in the paper). On a subset of SuperGLUE, in the zero-shot and one-shot settings, the BLOOM authors report that mGPT-1.3B has comparable performance despite having fewer weights than the other models in that comparison. One-shot, it performs on par with GPT-J-6B §4.4 fig. 6. A first-place system in a multilingual clause-level morphology shared task used mGPT-1.3B with prefix-tuning §4.4. This is reported by the paper, not verified here.

Generation diversity (story generation, 100 samples per language). Setup: temperature 1, max length 100, top-k 5, top-p 0.9 §4.5. Values are as printed tab. 11.

tab. 11 §4.5

The paper says Chinese has the highest diversity scores and the shortest mean length, which it attributes to its logographic writing. It reports English as lower than the other languages on these metrics, with longer mean length §4.5.

Monolingual continued-pretraining models. The paper releases 23 mGPT-1.3B variants continued on monolingual data. Each is listed with its held-out perplexity tab. 12 §5.2. The normalisation for these perplexities is not stated.

tab. 12 §5.2

Compute and emissions (estimated by the authors). The paper estimates CO2 with the formula in eq. 2 §7.2.

CO_2 = \frac{PUE \cdot kWh \cdot I^{CO_2}}{1000} eq. 2

§7.2 eq. 2

Paper vs code

The official repository is github.com/ai-forever/mgpt, pinned at ae05358c5d9537ecf8667bf010b70777370d3e77. The paper's footnote links it §1. The repository contains evaluation and inference code, notebooks and an example. It does not contain pretraining, tokenizer-training or data-processing code. The checks below compare the paper with the code that is there.

• License. The README badge says MIT [README.md#L4]. LICENSE.txt instead says the repository is sublicensed under a "Public Resource License" and rewrites the MIT text [LICENSE.txt#L1-L3] [LICENSE.txt#L93]. This is not a standard license, and it contradicts the README. The paper's abstract says MIT (abstract). Treat the code license as unclear. No code is copied verbatim here, so no snippet depends on this.
• Model hub. The paper's footnote points to ai-forever on Hugging Face §1. The code's usage example and notebooks load sberbank-ai/mGPT [README.md#L25] classification.ipynb. Both names appear in the repository, and it is not stated which is current.
• Few-shot protocol. The paper uses 0, 1, 4 and 16 shots, with five seeds for few-shot runs tab. 7 §4.2.1. The classification notebook sets shots_list = [0, 1, 2] classification.ipynb. The few-shot demonstrations are drawn with np.random.choice mgpt_classification.py, and no seed is set in the files shown.
• Hate-speech decision rule. The paper's Method describes one prompt per label and the lowest summed NLL §4.2.1. Table 6 lists five words per polarity tab. 6. The notebook scores ten prompts, five negative and five positive, and takes the argmax of summed token log-probabilities over all ten. The winner's polarity is the label hate_speech.ipynb hate_speech.ipynb. This equals a best-prompt-per-label rule. The paper does not say how the five words are combined.
• Few-shot truncation. The classification path keeps the last tokens when a prompt exceeds the context length mgpt_classification.py. The sequence-labelling path has no such step. Its dataset truncates on the right with truncation=True list_docs_dataset.py. A long demonstration block could therefore cut off the candidate tag. This is an inference from the code and was not run.
• Classification templates. The XNLI template in the code adds a comma after the label, which tab. 6 does not show mgpt_classification.py. The non-English verbalizers are hand-listed per language mgpt_classification_configs.py. The paper says that non-English prompts were machine-translated, but this sentence is under the XGLM baseline paragraph §4.2.1. The source of the code's translations is not stated.
• Task coverage. The code defines an AMAZON review-rating task with six languages mgpt_classification_configs.py. The paper does not report it tab. 7.
• Language sets. The code lists per-task languages: XNLI 15 mgpt_classification_configs.py, PAWSX 7 mgpt_classification_configs.py, XCOPA 11 mgpt_classification_configs.py, XWINO 4 mgpt_classification_configs.py. The paper does not list the languages averaged in tab. 7.
• Sequence-labelling seed. classify_with_examples takes a seed=1337 argument that the function body does not use sequence_labeling_prediction.ipynb. Demonstrations come from random.sample, so runs are not reproducible from the code as shown.
• Generation settings. The paper's footnote gives temperature 1, max length 100, top-k 5 and top-p 0.9 §4.5. The README's general setting is top-k 0, top-p 0.8, no-repeat n-gram 4, and its English setting is top-p 0.95 with top-k 0 [README.md#L45-L58]. The Hugging Face example uses top-k 10 and top-p 0.0 mgpt_huggingface_generation_example.ipynb.
• Perplexity. Eq. 1 normalises by characters eq. 1 §3.2. The validation example computes the exponential of the mean of per-batch mean token losses, so it is token-normalised mgpt_validation_example.ipynb.
• Padding and truncation utility. strip is ignored in the dataset helper because the stripped list is assigned to a misspelled variable, so the flag does nothing list_docs_dataset.py.
• Context length in code. The evaluation default context is 512 tokens mgpt_classification_configs.py, which matches the pretraining context in the paper §3.4.
• Batch and training settings. The README repeats the paper's batch size 2048, context 512, 600k steps and 400B tokens [README.md#L77]. No training code backs these numbers.

Pitfalls

• Perplexity normalisation conflicts. Section 3.2 normalises by characters eq. 1 §3.2, but the language-modelling section normalises by tokens §4.1. The validation example also normalises by tokens mgpt_validation_example.ipynb. Tokenizer perplexities in tab. 3 and language-level perplexities in fig. 3 may therefore not be comparable. Eq. 1 also sums from i = 0 to |t| and writes the log term as a malformed subscript. The card writes the log term as \log p_\theta(x_i | x_<i) instead of the printed log_{p_θ}. Whether the sum starts at 0 or 1 is not stated.
• Context window is stated three ways. The architecture is said to reproduce GPT-3 §3.3, and GPT-3 uses 2048 tokens. The ablation used 2048 §3.2. The final pretraining used 512 §3.4. The final value is 512.
• Token count does not match the batch setting (derived). A batch of 2048 sequences of 512 tokens for 600k steps gives about 6.3 x 10^11 token positions, while the paper states 400B tokens §3.4. Padding, packing or sampling is not described, so the gap is unexplained.
• CO2 power value looks wrong (derived). The text gives 91.3 kWh for mGPT-13B §7.2. With PUE 1.3 and 400 g per kWh, the printed 47.5k kg CO2 implies 91.3k kWh eq. 2. The table above keeps the printed value and flags it.
• Final-model training details are missing. Not stated: learning rate, optimizer, schedule, warmup, precision, attention head count, final vocabulary size, the sparse-attention pattern, and per-language sampling weights §3.1 §3.3 §3.4. The ablation learning rate (2e-4) and batch size (4) do not describe the final runs §3.2.
• Tokenizer choice rests on a small ablation. The five variants were 163M-parameter models with batch size 4 and context 2048 §3.2. The default strategy wins on perplexity only, and the authors note that perplexity captures only part of generalisation §5.4.
• Entropy wording conflicts with the compression filter. Section 3.1 gives a zlib ratio range of ×1.2 to ×8 and then describes thresholds in entropy terms §3.1. The code for this filter is not in the repository, so the filter cannot be reproduced from the code.
• Table 7 averages are not defined. The languages averaged in each column are not listed. The code's per-task sets differ from the full 61-language list mgpt_classification_configs.py.
• Chinese is evaluated but not in the 61 languages. Mandarin appears as a comment in the language-list table source tab. 1. The code includes zh in XNLI, XCOPA, AMAZON and POS configs mgpt_classification_configs.py sequence_labeling_prediction.ipynb. The 61-language count therefore does not describe every evaluated language.
• Baselines are not like-for-like. NER and POS baselines are fine-tuned and mGPT is 4-shot §4.2.2. The XGLM baselines use reproduced prompts and some results did not reproduce §4.2.1.
• Correlation claim outruns the table. The text describes a positive correlation between corpus size and perplexity for some languages §4.1. Table 5 reports a Pearson p-value of 0.137 for mGPT-1.3B, and no coefficient is given tab. 5.
• Abstract says "on par" with XGLM. The few-shot results in tab. 7 show mGPT-13B below XGLM-7.5B on most tasks. Use the table, not the summary.
• Monolingual Bulgarian link. tab. 12 prints the Belarusian model path for Bulgarian. The README lists mGPT-1.3B-bulgarian [README.md#L88]. Check the path before use.
• Sparse attention is not specified. The paper says the layers alternate dense and sparse attention §3.3. Which layers, and the sparse pattern, are not stated.
• Table and figure numbers are from source order. No compiled PDF was used. The numbers follow the order of captions in the LaTeX source, so check them against the published version before citing a number in another paper.
• Figures are described, not reproduced. Language-level perplexities, family-level perplexities, MLAMA probing and SuperGLUE results appear only as figures fig. 3 fig. 4 fig. 5 fig. 6. Numbers taken from them are not in this card.
• No appendix content. The source has an \appendix command with no content, so [app. ...] citations do not apply.

Lineage

• Builds on: GPT-3 for the architecture and the few-shot approach §1 §3.3 (no card on this wiki); the GPT-2 code base from Hugging Face §3.3; Megatron-LM and DeepSpeed for training §3.4; the Vowpal Wabbit classifier for data filtering §3.1.
• Compared against: XGLM-1.7B and XGLM-7.5B, the main baseline §4.2.1 tab. 7 (no card on this wiki); XLM-R base, M-BERT and Unicoder for NER and POS tab. 9 tab. 10; BLOOM-176B, OPT-175B, GPT-J-6B and T0-11B in the external SuperGLUE comparison by the BLOOM authors §4.4 fig. 6.
• Superseded by: not determined from this wiki. The paper notes that the Chinchilla scaling laws appeared after its work §5.1.