No Language Left Behind: Scaling Human-Centered Machine Translation

In plain words

Most machine translation systems cover a small set of widely spoken languages. Many languages have little digital text and few human-made translations to learn from, so their translations are poor or missing. This work tries to cover more than 200 languages with one model. The team first talked with native speakers of low-resource languages to learn what they need from translation tools. They then built test sets and training data for those languages. Some of the training data is paid for by hand, some is found by searching the web for sentences that match across languages, and some is created by translating monolingual text with earlier models. Professional translators write the test sets and part of the seed training data.

The model itself is one large network that is split into many "experts". For each word, only a couple of experts run, so the model is large but each word is cheap to process. The authors add regularisation against overfitting on small language sets, and filters against hallucinated, harmful output. They check the results with automatic scores and with human raters.

TL;DR

• One many-to-many translation model, NLLB-200, covers 202 languages with a 54.5B-parameter sparse Mixture-of-Experts Transformer §8.2.4.
• Training data combines human-translated seed bitext, mined bitext, and backtranslated data, with a data pipeline that filters noise §8.1.2–§8.1.4.
• Key training choices are expert output masking (p = 0.2) and a four-phase curriculum that adds low-resource pairs late §6.2.2, §8.2.3.
• Author-reported: on FLORES-101 the model beats the prior best by +7.3 spBLEU on average, a 44% relative gain §8.3.1, tab. 30.
• Code: the paper links the nllb branch of fairseq §1, fn. 1. Its final training script sets 170k updates, not the 300k in the paper; see Paper vs code.

Problem

Machine translation is concentrated on a few languages. Models for hundreds of languages are hard to build because most of them have very little parallel data §1. Existing multilingual work mostly covered about 100 languages, and evaluation for the larger extensions was weak §6. The earlier benchmark FLORES-101 covers about 100 languages, so more languages needed evaluation data too §1.

The authors count a language as low-resource when it has fewer than 1 million publicly available example sentence pairs §2.1. Under this definition 150 of the 204 languages in the benchmark are low-resource §3.

Three problems have to be solved together: finding human-quality evaluation and training data for low-resource languages, training a single model without interference between unrelated language directions, and keeping output safe when models hallucinate on little data §1, §6, §7.3. The paper adds a human-centred study with 44 native speakers of 36 low-resource languages to shape these goals §2.1.

Method

Overview

The pipeline has four parts §1, fig. 2:

1. Human-translated data: FLORES-200 for evaluation, and NLLB-Seed plus NLLB-MD as professional training and domain data §4.
2. Automatically created training data: language identification, monolingual data collection, and bitext mining with LASER3 §5.
3. A sparse Mixture-of-Experts translation model trained with regularisation and curriculum on the combined data §6, §8.2.
4. Evaluation with automatic metrics and human judgement, plus toxicity checks §7.

The sections that define the final recipe are §8.1 (data) and §8.2 (model).

Step 1: languages and evaluation benchmark

FLORES-200 has 3001 sentences taken from English Wikimedia projects, and is translated into 204 languages §4.1.1. Each language went through an alignment step, translation by professional translators, automatic checks, and review by independent reviewers. A language is accepted when the review scores above 90% §4.1, fig. 4. The benchmark gives 40,602 translation directions for the 202 trained languages §8.3.2.

Step 2: training data

• Primary bitext: 661 sets of public parallel corpora, English-centric where available, plus non-English-centric pairs where one side is low-resource §8.1.2, tab. 52. NLLB-Seed contributes parallel data for 39 languages §1, tab. 52.
• Mined bitext: sentence pairs found by bitext mining with LASER3 sentence encoders §5.3. Mined data covers 784 directions. A direction is kept if its xsim error rate is under 5 §8.1.2.
• Monolingual text: 192 languages, used for backtranslation and for self-supervision experiments §8.1.2.
• Backtranslation: the authors use two generators. A multilingual neural model (MMTBT) and per-pair statistical models (SMTBT) §8.1.3. Directions are chosen with a 3.3B dense baseline: SMTBT for languages whose baseline spBLEU is below 10 (eng_Latn-xx) or 15 (xx-eng_Latn), and MMTBT for directions below 30 spBLEU. This gives 76 SMTBT and 261 MMTBT directions §8.1.3.

Together the final dataset covers 1220 language pairs, 2440 directions, and about 18B sentence pairs §8.1.6, fig. 35. Two languages from the benchmark, arb_Latn and min_Arab, have no training data §8.1.6, fn. 36.

Step 3: filtering

The filters are applied in this order §8.1.4:

• LASER score: mined pairs with a score below 1.06 are removed §8.1.4.
• Length ratio: pairs with a length ratio above 9.0 are removed for mined and backtranslated data. Backtranslated sentences shorter than 15 characters are also removed §8.1.4.
• Length scaling: lengths are normalised by a per-language factor α_ℓ = N_eng_Latn / N_ℓ, computed from the FLORES-200 validation set §8.1.4.
• Language identification: pairs whose sides are not in the expected language are dropped §8.1.4, §5.1.
• Toxicity: pairs with a large difference in toxic items between source and target are dropped §8.1.4, §7.3.3.
• Deduplication: pair duplicates, source duplicates (for backtranslation) and target duplicates (for mining) are removed after normalisation §8.1.4.

Filtering improves bilingual models trained on the filtered data and reduces hallucinated toxicity §8.1.4, tab. 27.

Step 4: tokenizer

One SentencePiece model is trained for all languages on 100M sentences, with temperature sampling at T = 5 to upsample low-resource languages §8.1.1. The vocabulary has 256,000 entries §8.1.1. Three languages (zho_Hans, zho_Hant and yue_Hant) had high unknown-token rates, so they are upsampled by a factor of 5 during tokenizer training. After that the unknown-token rate is below 1% for every language §8.1.1.

Step 5: model architecture

The model is a Transformer encoder-decoder §6.1.1, eq. 2–3. Each layer uses pre-LayerNorm with residual connections §6.1.1, eq. 4–5. The encoder receives the source language token as a prefix. The decoder receives the target language token as its first input §6.1.1. The final model has 24 encoder and 24 decoder layers, model dimension 2048, FFN dimension 8192 and 16 attention heads §8.2.4. In every fourth Transformer block the FFN is replaced by a sparse MoE layer with 128 experts §8.2.4, §8.2.2. Each token is sent to its top 2 experts §6.2. The total is 54.5B parameters, and the FLOPs are similar to a 3.3B dense model §8.2.4.

MoE layer definition (paper's notation):

# Sparse MoE sublayer with top-2 routing  [§6.2] [eq. 6] [eq. 7] [eq. 8]
# Input:  tokens (T, d); E expert FFNs; gate matrix W_g (d, E)
# Output: mixed tokens (T, d)

def moe_sublayer(tokens, gate_matrix, experts):
    # Routing probabilities over experts  [§6.2] [eq. 7]
    gates = softmax(tokens @ gate_matrix)           # (T, E)
    # Keep the two highest-scoring experts per token  [§6.2] [eq. 7]
    gates = top2_mask(gates)                        # (T, E), 2 nonzero per row
    # Weighted sum of the chosen experts' outputs  [§6.2] [eq. 8]
    outputs = zeros_like(tokens)                    # (T, d)
    for each expert e:
        outputs += gates[:, e] * experts[e](tokens)
    return outputs

The expert itself is a two-layer ReLU network §6.2, eq. 6. A load-balancing loss is added to the training objective, weighted by 0.01 §6.2, eq. 9.

Step 6: regularisation and curriculum

Expert output masking (EOM). For a random fraction p_eom of tokens, the outputs of the first and/or second expert are masked to zero during training §6.2.2, fig. 18. The final model uses p_eom = 0.2 §8.2.4. The paper compares EOM with overall dropout, final output masking, conditional MoE routing and gating dropout. EOM and CMR are the two best, and the authors choose EOM because CMR costs 18–25% more training time [§6.2.2, tab. 16, §8.2.2].

Phased curriculum. Language pairs are split into four buckets, and each bucket enters training at a fixed update. The start points are chosen so that pairs that overfit early join late §8.2.3:

# Phased curriculum over language-pair buckets  [§8.2.3]
# Buckets b0..b3 are ordered from earliest to latest entry.
# Total training length T = 300000 updates.
# Input:  update index; buckets (list of language pairs per bucket)
# Output: language pairs active at this update

def active_pairs(update, buckets):
    # Entry offsets k = (300k, 130k, 70k, 30k): bucket b_i enters at T - k_i
    entry_updates = (0, 170000, 230000, 270000)
    active = []
    for bucket, entry in zip(buckets, entry_updates):
        if update >= entry:
            active += bucket
    return active

The bucket sizes are 822 pairs (673 low-resource) in b0, 506 (449) in b1, 455 (444) in b2 and 657 (612) in b3 §8.2.3.

Step 7: optional distillation

For deployment the authors distil the large model into smaller dense students. Offline (sequence-level) distillation uses translations from a fine-tuned 1.3B teacher, and online (word-level) distillation uses the teacher's soft distribution §8.6.1–§8.6.2. For the 54B NLLB-200 model the authors use online distillation, because the offline route would need translations for tens of thousands of directions. They distil to dense models of 1.3B and 615M parameters, trained for 200,000 updates §8.6.3. Results are in the Results section.

Hyperparameters

Values in the first block are from the paper. "Not stated" means the paper text read for this card does not give the value. Code values come from the nllb branch at code_commit. Config values use the names in the code.

Final model (NLLB-200)

Ablation setup (MoE vs dense study)

These settings are for the ablation experiments in Section 6. They are not the final recipe.

Distillation

Results

Author-reported. Setup: FLORES-200 devtest for the main results and FLORES-101 devtest for the comparison with prior work §8.3, tab. 30, tab. 33. The metrics are spBLEU with the SentencePiece-200 tokenizer and chrF++ §8.1.1, §8.3. Ablation numbers use the FLORES-200 dev set [tab. 16, tab. 28, tab. 29].

Comparison with prior work on FLORES-101 (devtest)

tab. 30 Full FLORES-101 devtest, 10k directions. Scores are spBLEU / chrF++. DeltaLM scores are from the FLORES-101 leaderboard. M2M-100 and Deepnet averages cover only the 87 languages that overlap with FLORES-101.

The author text puts the average gain at +7.3 spBLEU over the nearest prior system on 101 languages (44%), and +6.9 on the 87-language subset §8.3.1.

Main results on FLORES-200 (devtest)

tab. 33 NLLB-200 averaged over the directions in each group. Supervised pairs are the 2440 directions with training data; zero-shot pairs are the rest.

Directions: 201 eng_Latn-xx, 201 xx-eng_Latn and 40,200 xx-yy, for 40,602 in total. 38,162 of them are zero-shot §8.3.2. The text reports 39.5 chrF++ for the supervised xx-yy set; the table shows 39.7 §8.3.2, tab. 33.

Comparison with commercial systems

tab. 34 102 low-resource directions for which Google Translate is also available. Scores are spBLEU / chrF++.

Regional comparisons

• African languages: NLLB-200 beats MMTAfrica and M2M-100 (fine-tuned on MAFAND) on most directions tab. 31. MMTAfrica is slightly better on eng_Latn-yor_Latn and eng_Latn-swh_Latn §8.3.1.
• Indian languages: NLLB-200 beats IndicTrans and IndicBART on all xx-eng_Latn directions. On eng_Latn-xx it beats IndicTrans and IndicBART but not Google Translate or Microsoft Translate tab. 32, §8.3.1.

Ablations (dev set, chrF++)

tab. 28 Frequency of MoE layers (128 experts, FLORES-200 dev).

tab. 29 Curriculum strategies (128 experts, f_MoE = 4, T = 300k).

tab. 16 Regularisation strategies on the ablation set, MoE-64 (validation chrF++; † = best of its sweep).

The paper's final model uses EOM at p_eom = 0.2, which is not one of the swept values shown in this row §8.2.2, §8.2.4.

Distillation

tab. 40 Average FLORES-200 devtest chrF++ for the Wikipedia-domain models.

For the 54B model distilled to dense 1.3B and 615M students, the text reports +0.5 chrF++ over the dense 1.3B baseline and +0.3 chrF++ over the 615M baseline on average §8.6.3.

Human evaluation and toxicity

Human evaluation of translations §7.2 and the toxicity benchmark and results §7.3 are reported in the paper. They are not summarised in this card.

Paper vs code

Code: the official repository is facebookresearch/fairseq (MIT), and the code used here is its nllb branch, fetched at code_commit. The paper's footnotes link that branch §1, fn. 1, its data folder app. E.1, fn. 49 and the stopes library §8.1.4, fn. 35. Searched for an official repository: these paper links only. Code paths below are relative to the repository root at code_commit.

Differences found on reading the code (not run):

• Training length. The paper says the final model trains for 300k steps with four phases §8.2.4. The final script sets max_updates=170000 run_flores200_4.4_final.sh, while its phase string mu170.230.270 sets the entry points of phases 2–4 at 170k, 230k and 270k run_flores200_4.4_final.sh. Those entry points follow the paper's k values with T = 300k §8.2.3. The base config sets max_updates: 300000 flores_200_full_moe.yaml. The script and the config disagree, and the paper follows the config.
• Curriculum file name. The script's language-pair file is named final_lang_pairs_cl3.txt run_flores200_4.4_final.sh. It is a single line of pairs with no bucket markers final_lang_pairs_cl3.txt. I did not trace how the phase string maps pairs to buckets, so the bucket assignment is not verified.
• Second-expert routing. The paper says the second expert is chosen by top score without randomisation §6.2. The NLLB sweep sets all sweep_mmt.py, which is deterministic, so the two agree. The library default is sampling transformer_config.py.
• Load-balancing weight. The paper uses 0.01 §6.2. The NLLB sweep sets 0.01 sweep_mmt.py. The criterion's own defaults are 0.0 moe_label_smoothed_cross_entropy.py and 1.0 fairseq_criterion.py.
• Generation capacity. The paper sets generation capacity to the batch size §8.2.4. The NLLB config sets the evaluation capacity fraction to 1.0, so all tokens fit default.yaml. The gate's own default is 0.25 top2gate.py.
• Matches. The following agree between the paper and the final script or config: dropout 0.3, p_eom 0.2, MoE every 4th block, 128 experts, peak LR 0.002, warmup 8000 steps, label smoothing 0.1, Adam β and ε, attention dropout 0.1, pre-LN, and the 2 × T/E training capacity run_flores200_4.4_final.sh sweep_mmt.py top2gate.py.

Not checked in code: the exact load-balancing formula (eq. 9), and how EOM chooses between the first and second expert.

Pitfalls

• Training length conflicts between paper and script. The paper says 300k steps. The released final script stops at 170k. Check which one you need before comparing results. See Paper vs code.
• The default MoE routing is sampling. Set moe_second_expert_policy to all to match the paper transformer_config.py. The sampling variant changes which second expert is used.
• Gate-loss weight defaults differ. The criterion defaults are 0.0 and 1.0. Set the weight to 0.01 explicitly.
• The ablation LR is not the final LR. Ablations use a peak of 0.004 with the 1.3B dense baseline §6.2.1. The final model uses 0.002 §8.2.4. Do not transfer ablation numbers without rechecking.
• Ablation and final results use different sets. Ablations report the dev set tab. 16, tab. 28, tab. 29. The headline numbers use devtest tab. 33.
• Zero-shot is most of the benchmark. 38,162 of 40,602 directions were never trained on §8.3.2. Average scores hide large differences between supervised and zero-shot pairs.
• Two numbers disagree in the text. The text says 39.5 chrF++ for the supervised xx-yy set; the table says 39.7 §8.3.2, tab. 33. The text says a 515M distilled student; the table says 500M §8.6.2, tab. 40.
• Length normalisation depends on the evaluation set. The per-language length factor α_ℓ is computed from the FLORES-200 validation set §8.1.4. Reproducing the training data therefore depends on that evaluation data.
• Data comes from many public corpora under their own terms. The 661 primary corpora and the mined and backtranslated data are described in §8.1.2 and listed in tab. 52. Check each source's license before redistribution. The paper notes that scripts to recreate the training data are released §1.
• Serving the full model is expensive. The 54.5B-parameter model needs all experts in memory. The authors distil it to 1.3B and 615M dense students for cheaper use §8.6.3.
• Results are author-reported. The comparison with Google Translate is approximate: the authors note that commercial systems are proprietary and that few details about them are public §8.3.2.
• Hallucinated toxicity is a separate risk. Filtering reduces it on bilingual models §8.1.4, tab. 27. The toxicity benchmark and results are in §7.3 and are not covered here.

Lineage

• Builds on: Transformer encoder-decoder §6.1.1; top-k sparse gating following Lepikhin et al. (2020) §6.2; LASER3 sentence encoders §1, §5.3; FLORES-101 as the earlier benchmark (Goyal et al., 2022) §1, §4.1.
• Compared against: M2M-100, DeltaLM and Deepnet on FLORES-101 tab. 30; MMTAfrica and Mafand-MT for African languages tab. 31; IndicTrans and IndicBART for Indian languages tab. 32; Google Translate and Microsoft Translate tab. 32, tab. 34.
• Related regularisers in the paper: gating dropout (Liu et al., 2022) §6.2.2.
• Superseded by: not known from content already in this wiki.