The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

In plain words

Belebele is a reading test for language models in 122 languages. Each item is a short text, a question about it, and four possible answers, exactly one of which is correct. The model reads the text and picks an answer. People who read the English items score almost perfectly, and no model tested comes close.

Each question was written once in English and then translated, so the same items exist in every language and scores can be compared directly. The benchmark was made without machine translation. Experts translated the text and proofread it.

The main finding is that large models trained mostly on English do well in English and in several other languages, but drop off in many lower-resource ones. Smaller models pretrained on balanced multilingual text hold up better across the long tail of languages.

TL;DR

Belebele is a test set, not a training set. It contains 900 four-choice reading questions per language, parallel across 122 language variants, and its English items are already hard for models. The paper's own results put LLaMA 2 70B at 90.9% on English but 48.0 on average across languages. Multilingual masked models fine-tuned on translated data score higher on average. The benchmark is CC BY-SA, but the English training set is CC BY-NC and cannot be used commercially.

Problem

Most multilingual reading-comprehension benchmarks cover around 30 languages §1. Span extraction is sensitive to morphology and is hard to scale across languages, which is why multiple-choice questions were chosen §3.1. Without parallel items, scores in different languages cannot be compared directly §3. The authors also note that pretraining-corpus documentation is inconsistent, which limits what can be explained from the results [Limitations].

Method

The paper contributes a dataset and evaluation protocols, not a new model. Each item is built in three stages.

Writing the English questions. Annotators wrote multiple-choice questions with a language service provider in five iterations, with feedback after each round §3.1. The guidelines require that the correct answer be unambiguous, that wrong answers be plausible, and that questions be answerable from the passage alone. They also ban double negatives §3.1 app. A.5.1.

Quality checks. Each iteration was checked by manual inspection of a sample and by automatic lexical-overlap features §3.2 app. A.6. The heuristic filters removed about 20% of the final-iteration questions §3.2. The final set passed a t-test on those features (p = 0.81), where MCTest failed (p < 0.01). A logistic regression on bag-of-words features reached 0.28 accuracy on the 900 questions, against 0.25 for chance §3.2.

Translation. Passages come from the FLORES-200 corpus, which was itself translated by native speakers §2.4 §3.3. Translators were given the translated passage and asked to align the questions to it, even where the translation is ambiguous §3.3 app. A.5.2. For five Indo-Aryan languages (Hindi, Bengali, Urdu, Nepali, Sinhala), annotators also produced Latin-script versions with the help of IndicXlit §3.3.

Evaluation protocols. Four settings are used §4.2:

• Fine-tuning a masked model with a multiple-choice head, either on the English training set (zero-shot cross-lingual transfer) or on machine-translated training data for all target languages (Translate-Train-All) §4.2 app. A.4.1.
• Five-shot in-context learning: five English examples from the training set are prepended. The answer is the letter with the highest probability §4.2 app. A.4.2.
• Zero-shot instructions in English, with the letter answer parsed from the output §4.2 app. A.4.3.
• Translate-test: passages, questions and answers are machine-translated back to English before prompting §4.2 §5.1.

The five-shot protocol, as described in the paper:

# Five-shot in-context scoring of one language  [app. A.4.2] [§4.2]
# Input:  model; shots: 5 English training items; rows: target-language items
# Output: accuracy in [0, 1]; the paper averages 3 runs

def format_item(item, with_answer):
    # Template: passage, question, four lettered options, then the answer
    # [app. A.4.2]
    text = "P: " + item.passage + "\n"
    text = text + "Q: " + item.question + "\n"
    for letter, option in zip("ABCD", item.options):
        text = text + letter + ": " + option + "\n"
    if with_answer:
        text = text + "Answer: " + item.correct_letter + "\n"
    else:
        text = text + "Answer:"
    return text


def score_five_shot(model, shots, rows):
    # Prefix the prompt with the English examples  [app. A.4.2]
    prefix = "".join(format_item(shot, True) for shot in shots)
    correct = 0
    for row in rows:
        prompt = prefix + format_item(row, False)
        # Pick the letter with the highest probability relative to the
        # other three  [app. A.4.2]
        scores = [model.logprob(prompt, letter) for letter in "ABCD"]
        prediction = "ABCD"[scores.index(max(scores))]
        if prediction == row.correct_letter:
            correct += 1
    return correct / len(rows)

The paper's own ICL template uses "P:", "Q:" and "Answer:" labels; the zero-shot prompt uses a different, "###"-separated layout app. A.4.2 sample_zero_shot_instructions.md. The pseudocode follows the ICL template.

Hyperparameters

Fine-tuning settings for the runs reported in tab. 5 and tab. 6. Values are as printed in the paper.

Human baseline: four authors each answered about 30 randomly sampled English items blind, averaging 97.6 §5.1. The 95% CI for all 900 questions is [93.1, 99.5] §5.1.

Results

Author-reported. Setup: the 900-item Belebele test set in 122 variants §3.5; models are the MLMs, LLMs and instruction-tuned models listed in §4.1 tab. 2; the baseline is chance (0.25) for classifiers §4. Hardware is not stated.

Summary across all 122 variants tab. 2. "%≥50" and "%≥70" give the share of languages above 50% and 70%.

** BLOOMZ was fine-tuned for translation on FLORES-200, so the paper flags its zero-shot result as an unfair advantage and does not analyse it tab. 2 §4.1 [Ethics Statement].

Translation-based zero-shot results for LLaMA 2-chat 70B tab. 3:

Other reported figures, all author-reported:

• English difficulty: the best full fine-tune in English is RoBERTa-base at 71.7 §5.1; LLaMA 2 70B 5-shot reaches 90.9 on English §5.1 tab. 2. LLaMA 1 7B reaches 37.3 §5.1.
• All models evaluated underperform the human sample §5.1.
• The XLM-V, InfoXLM and XLM-R Translate-Train-All scores are about 10 points below their reported XNLI Translate-Train scores. Across 15 languages and three models, the correlation with XNLI is r = 0.85 §5.1.
• LLaMA 2 70B "largely outperforms" XLM-R on high-resource languages but reaches 50 accuracy on about half as many languages §5.2 tab. 2. LLaMA 2-chat (70B) is above 35 accuracy for 59 languages §5.2.
• Translate-Test beats in-language performance for most of the 91 evaluated languages. Only German and Italian are better in-language, while 68 are better when translated to English, none of them high-resource §5.2 app. A.7.4 tab. 3.
• XLM-V (902K vocabulary) outperforms XLM-R and InfoXLM (250K vocabulary) on low-resource languages with the same architecture and training data (CC-100) §5.2.
• LLaMA 1 7B is slightly above chance in English. Only the 30B and 65B checkpoints are non-trivial on Japanese and Greek, language families the paper says are not in the LLaMA 1 pretraining corpus §5.2 fig. 4.
• The masked models are pretrained on multilingual corpora of about 100 languages. LLaMA and Falcon are English-centric, with LLaMA 1 reportedly using under 4.5% non-English text §4.1.
• Native script beats Latin-script romanization for all models except Falcon, on the six languages compared §5.2 app. A.7.3.
• Machine-translated instructions fall below chance for about 25 languages §5.2.

Paper vs code

The repository (commit 918890beb2290a8d3ef2d7a90369925959e1bacf) contains the training-set assembly script and a sample of the zero-shot prompt and parsing code. It contains no model training code, no evaluation harness, and not the benchmark data (downloaded separately [README]). The paper's full harness is therefore not checkable.

• Development set. The paper says the development set uses "the respective validation sets of the 6 above datasets" app. A.2. The script reads only the train split for MCScript2.0 and says so in a comment [code: assemble_training_set.py#L198-L199].
• Passage length filter. The paper says it filters "excessively long passages" app. A.2 without a number. The code keeps passages of at most 25 sentences for SciQ and MultiRC and for MCScript2.0 [code: assemble_training_set.py#L113] assemble_training_set.py assemble_training_set.py.
• Fill-in-the-blank and multi-answer filtering. Consistent with the paper. The SciQ script drops questions containing "_" [code: assemble_training_set.py#L76]; the MultiRC script keeps only questions with exactly one correct answer and at least four options [code: assemble_training_set.py#L154-L156].
• Random seed. The script fixes NumPy's seed to 0 before shuffling answers assemble_training_set.py. The paper does not mention a seed.
• Zero-shot answer parsing. The paper says it "accepts multiple formats" app. A.4.3. The sample code strips parentheses and accepts the letters A–D. If the output is not one of those letters, it takes the first character of the output as the answer sample_zero_shot_instructions.md. That fallback is not described in the paper.
• Prompt format. The paper says the instruction format "did change slightly across the models" app. A.4.3. The sample file confirms this and gives one template, not the ICL template sample_zero_shot_instructions.md sample_zero_shot_instructions.md.
• Footnote on code. The paper points to the repository for the zero-shot instructions and post-processing code app. A.4.3. The repository has only the sample file, not the full code used for the reported runs.

No discrepancies found in the data-filter logic beyond the points above.

Pitfalls

• Test set, not training set. The authors say Belebele is for testing only §3.4. Using it for training or tuning contaminates the benchmark.
• The training set is non-commercial. The English training set and its assembly code are assembled from six datasets. The majority is CC BY-NC app. A.3. The repository says the same [README]. Commercial fine-tuning on this training set is not permitted, and the paper says it needs a different data source app. A.3. Do not reproduce the training set for commercial use.
• Dataset license. The benchmark is CC BY-SA, like the FLORES-200 passages app. A.3. Derived datasets carry the share-alike term.
• Translate-Train-All is not reproducible exactly. The machine-translation system is not named §4.2 app. A.4.1. The data is capped at 650k samples for one epoch, and the selection is not described beyond that app. A.4.1.
• Printed learning-rate typo. The XLM-V Translate-Train-All learning rate is printed as "3-6" tab. 6. Do not infer a value; the paper does not state it.
• Human baseline is a sample. The 97.6 figure comes from four authors answering about 30 English items each §5.1. The CI is given for the full 900 items §5.1.
• Chance level differs by protocol. The 0.25 chance level applies to classifiers §4. The paper notes that exact-match, sequence-to-sequence evaluation does not have this lower bound §4.
• Answer parsing is lenient. The zero-shot fallback in the sample code counts any output that starts with a letter A–D as that answer [code: sample_zero_shot_instructions.md#L40-L45]. Report parse-failure rates when you use it.
• Script count inconsistency. The introduction says seven languages appear in two scripts §1. Section 3.3 names five Indo-Aryan languages plus Modern Standard Arabic, which is six §3.3. The language table is the reference tab. 4.
• Translationese and errors. Translations can be literal, so the same question may be harder in some languages [Limitations]. Annotators found quality issues in the FLORES translations, and the authors changed questions to fit the translated passages rather than correcting FLORES [Limitations].
• GPT-3.5 cannot be audited. Its pretraining and fine-tuning data are not disclosed, and the paper says cross-model comparisons with it may be unfair [Limitations] §4.1.
• Table and figure numbers here follow the order of the LaTeX source, not a compiled PDF. Check them against the PDF before citing.

Lineage

• Builds on: FLORES-200 passages and translations §2.4 (Goyal et al. 2022, NLLB 2022; not in wikillm). Question construction follows the protocols cited in §3.1.
• Compared against: XNLI §5.1, and MCTest for the quality checks §3.2.
• Training set built from RACE, SciQ, MultiRC, MCTest, MCScript2.0 and ReClor app. A.2. None of these has a card here, and only their names are used.
• Evaluated models: XLM-V, arXiv:2301.10472, LLaMA, arXiv:2302.13971, LLaMA 2, arXiv:2307.09288 §4.1. XLM-R and InfoXLM are also evaluated §4.1. Those papers have no cards here.
• Superseded by: not known from content already in this wiki.