---
kind: concept
slug: multilingual-evaluation
title: Multilingual evaluation
summary: >-
  Measuring how models behave across languages and cultures. Belebele gives parallel items in 122 variants. Other designs score many-language tasks by label likelihood, probe Arab versus Western entity preference, simulate survey respondents, test one country's answers, run code in five programming languages, or score word vectors on new pair lists. Each asks something different. Author-reported.
tags: [evaluation, multilingual, monolingual, benchmark, reading-comprehension, cultural-bias, code-generation]
papers: ["2004.09095", "2004.09813", "2101.01785", "2103.12028", "2204.07580", "2303.17466", "2303.17568", "2304.05613", "2305.14456", "2308.02976", "2308.16884", "2402.13231", "2402.13605", "2410.12985", "2609.34138"]
related: [llm-evaluation, benchmark-validity, in-context-learning, masked-language-modeling, language-model-pretraining, subword-tokenization, human-evaluation, distributional-alignment, prompt-engineering, sequence-labeling, llm-ethics, bibliometric-analysis, code-completion, dataset-curation, knowledge-distillation, text-embedding, zero-shot-learning, language-identification, neural-machine-translation, text-classification]
status: draft
updated_at: 2026-11-01
---

## Overview

Multilingual evaluation asks how well a model performs in each language it is
meant to serve, and whether the numbers can be compared. The invariant that
makes comparison possible is parallel data: the same items exist in every
language, so a score gap comes from the language or the model, not from
harder questions [arxiv:2308.16884 §3]. The authors note that most
multilingual reading-comprehension benchmarks cover around 30 languages
[arxiv:2308.16884 §1].

Belebele is the source here. It has 900 four-choice reading questions on 488
passages, in 122 language variants, so each language has 900 rows and the
dataset has 109,800 rows in total [arxiv:2308.16884 §3.5] [arxiv:2308.16884 tab. 1].
Questions are written in English first. Translators then render each passage
and align its questions to the translated text, even where the translation is
ambiguous [arxiv:2308.16884 §3.3]. The passages come from FLORES-200, which
native speakers translated [arxiv:2308.16884 §2.4] [arxiv:2308.16884 §3.3].

Multiple choice was chosen over span extraction. Span extraction is sensitive
to morphology and is harder to scale across languages [arxiv:2308.16884 §3.1].
The benchmark is a test set. The authors say it should not be used for training
[arxiv:2308.16884 §3.4].

Parallel items are not the only design in the cards. Other designs measure
different things, and their numbers should not be compared directly:

- **Label-likelihood suites** score a generative model on many tasks in many
  languages, with a few demonstrations [arxiv:2204.07580 §4.2].
- **Cultural-preference probes** check whether a model prefers culturally
  Western or Arab entities in Arabic text [arxiv:2305.14456 §4.4].
- **Survey simulation** asks a model to answer as a survey respondent in a
  given country and language [arxiv:2402.13231 §4.5].
- **Single-language suites** test a model pre-trained on one language against
  the best multilingual baseline the authors could find, on tasks in that
  language [arxiv:2308.02976 §4] [arxiv:2308.02976 tab. 1].
- **Resource-class audits** score no model. They sort languages into six
  classes by labelled and Wikipedia data, then measure how far NLP venues
  mention each class [arxiv:2004.09095 §2.2] [arxiv:2004.09095 §2.3].
- **Country-specific alignment** scores a model against one country's survey
  answers and school-curriculum questions [arxiv:2402.13605 §3] [arxiv:2402.13605 §4].
- **Multilingual code generation** runs one set of programming problems in
  several programming languages and scores each by unit tests
  [arxiv:2303.17568 §3.1] [arxiv:2303.17568 §3.2]. Here "multilingual" means
  programming languages, not natural languages, so these numbers do not
  transfer to the other designs.
- **Human-rated word-similarity sets** add a test list for a language that has
  none, then score word vectors by rank correlation with human ratings. Each
  list is monolingual and is built by translating two English lists, so its
  scores are per language [arxiv:2609.34138 §4.1] [arxiv:2609.34138 §2].
- **Distilled sentence encoders** train a multilingual student to match a
  fixed English teacher's vectors on translation pairs, then score cross-lingual
  similarity and bitext mining [arxiv:2004.09813 §2] [arxiv:2004.09813 §4.1].
- **Single-language benchmark suites** test one language across many tasks at
  once, here Arabic over 42 datasets [arxiv:2101.01785 §5.1].
- **Zero-shot chat model on existing task sets** sends one prompt per example
  to a chat model, with no demonstrations, across seven NLP tasks in 34 to 37
  languages. Each task uses its own existing dataset, not a shared parallel
  set [arxiv:2304.05613 §3] [arxiv:2304.05613 §1].
- **Human audit of web-crawled corpora** reads random lines from a multilingual
  training or release corpus and labels each one, with no model involved. It
  checks whether a subset contains the language its code claims
  [arxiv:2103.12028 §4.1] [arxiv:2103.12028 §4.2].
- **Survey probe across prompt languages** asks a chat model the same
  Hofstede culture-survey items in English with a country named, in the
  country's language, and in a second English wording [arxiv:2303.17466 §3.1].
- **Translate-then-classify** has one LLM translate English posts into Greek,
  then classify the translations for depression severity. It is scored against
  the labels of the English originals [arxiv:2410.12985 §3] [arxiv:2410.12985 §4].

## Variants

The first five variants are protocols for scoring the same parallel benchmark
(Belebele). Each is a protocol, not a different dataset [arxiv:2308.16884 §4.2].
The remaining variants use other designs.

### Fine-tune a masked encoder on English (zero-shot transfer)

Fine-tune a masked multilingual model with a multiple-choice head on the
English training set, then test it in every language without further training.
This is zero-shot cross-lingual transfer [arxiv:2308.16884 §4.2]
[arxiv:2308.16884 app. A.4.1]. Card: [The Belebele Benchmark](/papers/2308.16884).
Full fine-tuning of XLM-R, XLM-V and InfoXLM (large) averages 54.0 to 56.2
across 122 variants [arxiv:2308.16884 tab. 2].

### Fine-tune on machine-translated training data (Translate-Train-All)

Machine-translate the English training set into every target language and
fine-tune on the combined data, for one epoch with a cap of 650k samples
[arxiv:2308.16884 app. A.4.1]. This variant scores highest among the encoder
results: 58.9 to 60.2 average [arxiv:2308.16884 tab. 2]. The paper does not
name the machine-translation system [arxiv:2308.16884 §4.2]. Card:
[The Belebele Benchmark](/papers/2308.16884).

### Few-shot prompting in English (five-shot in-context learning)

Prepend five English examples from the training set to each test item. The
answer is the letter with the highest probability among A to D
[arxiv:2308.16884 §4.2] [arxiv:2308.16884 app. A.4.2]. The paper averages three
runs and does not state seeds [arxiv:2308.16884 app. A.4.2]. Card:
[The Belebele Benchmark](/papers/2308.16884).

### Zero-shot English instructions

Give an English instruction, the target-language item, and read the answer
letter from the generated output [arxiv:2308.16884 §4.2]
[arxiv:2308.16884 app. A.4.3]. Card: [The Belebele Benchmark](/papers/2308.16884).

### Translate-test

Machine-translate the passage, question and options into English, then use an
English prompt [arxiv:2308.16884 §4.2] [arxiv:2308.16884 §5.1]. For LLaMA 2-chat
70B, translate-test averages 57.1 over 91 non-English languages, against 44.1
for the same items in-language [arxiv:2308.16884 tab. 3]. Card:
[The Belebele Benchmark](/papers/2308.16884).

### Label likelihood with few-shot demonstrations (mGPT suite)

Score each candidate label, or each candidate tag for one word, by the summed
negative log-likelihood of the full prompt. Demonstrations come before the
example [arxiv:2204.07580 §4.2.1]. For sequence labelling, the model tags
words left to right, and each word's tag is chosen the same way
[arxiv:2204.07580 §4.2.2]. The suite covers classification (XWINO, PAWSX,
XCOPA, XNLI and a hate-speech set), NER and POS, and knowledge probing on
mLAMA in 23 languages [arxiv:2204.07580 §4.2] [arxiv:2204.07580 §4.3].
Card: [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580).

The authors compare two decoder-only models, about 1.3B and 13B parameters,
with XGLM-1.7B and XGLM-7.5B that they reproduced [arxiv:2204.07580 tab. 7].
On XWINO, mGPT-1.3B scores 56.2 zero-shot, against 54.2 for XGLM-1.7B
[arxiv:2204.07580 tab. 7]. Few-shot NER and POS scores are far below
fine-tuned encoders: 15.0 NER F1 and 24.4 POS accuracy for mGPT-1.3B at four
shots [arxiv:2204.07580 tab. 9] [arxiv:2204.07580 tab. 10]. The authors say
these baselines are not like-for-like, because the encoders are fine-tuned
[arxiv:2204.07580 §4.2.2].

### Masked-entity preference score (CAMeL)

Take a prompt in Arabic in which one culture-specific entity is replaced by a
mask. Compare the probability of Arab entities with the probability of Western
entities at that mask. The Cultural Bias Score is the share of (Arab, Western)
pairs where the Western entity wins, averaged over prompts. Zero is the
ideal [arxiv:2305.14456 §4.4] [arxiv:2305.14456 tab. 10]. Card:
[Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](/papers/2305.14456).

The score needs a per-token probability at the mask. Masked-language models
score the full prompt, while causal and T5-type models score only the text
before the mask [arxiv:2305.14456 §4.4]. The paper samples 50 Arab and 50
Western entities per entity type in each of five runs [arxiv:2305.14456 tab. 10].
Averaged over the Co prompts, scores run from 45.5 (AraGPT2 large) to 63.5
(BLOOM) [arxiv:2305.14456 tab. 10]. Prepending Arab demonstrations lowers the
score for most models, while a culture token has little effect
[arxiv:2305.14456 §4.4].

### Survey-persona alignment with hard and soft scores (AlKhamissi et al.)

Ask a model to answer as one real survey respondent, described by six
demographic fields: region, sex, age, social class, education and marital
status. Compare its answer with the answer the respondent gave in the World
Values Survey, wave 7 [arxiv:2402.13231 §4.1] [arxiv:2402.13231 §4.2]
[arxiv:2402.13231 tab. 1]. Each question has four paraphrases. The model is
sampled five times at temperature 0.7, and the majority answer is kept
[arxiv:2402.13231 §4.5]. Hard alignment is exact agreement. Soft alignment
gives partial credit on ordinal items [arxiv:2402.13231 §4.5] [arxiv:2402.13231 eq. 1].
Card: [Investigating Cultural Alignment of Large Language Models](/papers/2402.13231).

Scores are averaged over the 30 questions and 303 matched personas per country
[arxiv:2402.13231 §4.2] [arxiv:2402.13231 §4.5]. Averaged over the four models,
soft alignment is 47.2 on the Egypt survey and 59.1 on the US survey
[arxiv:2402.13231 tab. 3]. Hard alignment is 27.0 and 33.8 on the same surveys
[arxiv:2402.13231 tab. 3].

### Anthropological prompting on top of persona prompts

Keep the persona and question, and add a framework of seven points from
anthropological field methods. The model reasons in one paragraph under that
framework, then gives the option number [arxiv:2402.13231 §4.6]
[arxiv:2402.13231 app. I]. For GPT-3.5, English prompt and Egypt survey, soft
alignment rises from 0.483 to 0.510 and hard alignment from 0.244 to 0.284
against vanilla persona prompting [arxiv:2402.13231 tab. 4]. The paper runs
this on one model, one language and one survey, and does not report a
variance for it [arxiv:2402.13231 tab. 4].

### Monolingual encoder scored on a per-language task suite (BETO and GLUES)

Pre-train a BERT-Base-sized encoder on text in one language only, then fine-tune
and test it on a suite of tasks in that language. The suite, GLUES, covers
natural language inference, paraphrase, NER, POS tagging, document
classification and extractive QA [arxiv:2308.02976 §4]. The masking is
whole-word, and word-level tasks use a sliding window [arxiv:2308.02976 §3]
[arxiv:2308.02976 §5.1]. Card: [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976).

The baseline is the best multilingual BERT result the authors found in the
literature for the same task, not a run under their setup [arxiv:2308.02976
tab. 1] [arxiv:2308.02976 tab. 2]. Cased BETO scores 82.01 on XNLI and 98.97 on
POS. Uncased BETO scores 96.12 on MLDoc, against 95.70 for the baseline
[arxiv:2308.02976 tab. 1]. On TAR-to-XQuAD, the baseline leads on exact match,
61.80 against 57.06 for cased BETO [arxiv:2308.02976 tab. 2]. The authors
report these as their own numbers.

### Unit-test scoring across programming languages (HumanEval-X)

Write each benchmark problem by hand in every target language, with a
declaration, docstring and unit tests. The model writes the body from the
declaration and docstring. The answer is scored by running the tests, and the
metric is pass@k over n = 200 samples per problem, using the unbiased estimator
[arxiv:2303.17568 §3.1] [arxiv:2303.17568 §3.2] [arxiv:2303.17568 eq. 5]. The
translation task gives the model the source-language solution without its
docstring, across 20 ordered language pairs [arxiv:2303.17568 §3.2]. Card:
[CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](/papers/2303.17568).

The authors report average pass@1 of 18.40% on five languages, against 16.73%
for CodeGen-Multi-16B [arxiv:2303.17568 tab. 5]. The benchmark has 164
problems in five languages, so 820 pairs [arxiv:2303.17568 §3.1].

### Survey and curriculum scoring for one country (KorNAT)

Split the benchmark in two. Social-value items have no single correct answer:
the score for a chosen option is the share of a Korean survey that picked it.
Three metrics are defined, SVA, A-SVA and N-SVA, and the last one swaps in a
Neutral label for contested questions [arxiv:2402.13605 §3.1] [arxiv:2402.13605 §4].
Common-knowledge items come from Korean curriculum material and have one
correct option, scored by accuracy against a reference of 0.6
[arxiv:2402.13605 §3.2] [arxiv:2402.13605 §4]. Card:
[KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](/papers/2402.13605).

The survey weights each answer for age, sex, education, residence and income
[arxiv:2402.13605 §3.1.4]. On common knowledge, HyperCLOVA X (0.707), PaLM-2
(0.664) and Gemini Pro (0.639) are above the 0.6 reference, and the other four
models are below it [arxiv:2402.13605 tab. 3]. On social values, PaLM-2 has the
highest SVA at 0.331, against a maximum of 0.450 [arxiv:2402.13605 tab. 1].

### Human-rated word-similarity pairs (Indian-language sets)

Translate an English word-similarity list into the target language. Two
native speakers translate each pair, and a third breaks ties where they
disagree [arxiv:2609.34138 §4.1]. Then eight native speakers rate each pair
from 0 to 10, with guidelines that separate similarity from relatedness, and
the score is the mean of the eight ratings [arxiv:2609.34138 §4.2]. The source
lists are RG-65 and WordSim-353, and the six languages are Urdu, Telugu,
Marathi, Punjabi, Tamil and Gujarati [arxiv:2609.34138 §4.1] [arxiv:2609.34138 §2].
Card: [Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems](/papers/2609.34138).

To score a word-vector model, take the cosine of each pair and compute
Spearman's rho against the human means, times 100. The baselines are CBOW,
Skip-gram and FastText, trained on one monolingual corpus per language with
300 dimensions [arxiv:2609.34138 §2]. Baselines exist only for Urdu, Marathi
and Telugu, while Punjabi, Tamil and Gujarati have test sets alone
[arxiv:2609.34138 §7]. The FastText row with an OOV fallback beats plain FastText
in all three, at 45.47 against 34.61 for Urdu, 38.66 against 33.68 for
Marathi, and 46.02 against 34.29 for Telugu [arxiv:2609.34138 tab. 2]
[arxiv:2609.34138 tab. 3] [arxiv:2609.34138 tab. 4]. Marathi Skip-gram scores
41.22, above the OOV row [arxiv:2609.34138 tab. 3].

### Distilled student for cross-lingual sentence similarity (multilingual knowledge distillation)

Start from an English sentence encoder that is already good at similarity, and
use it as a fixed teacher. The student is a multilingual masked encoder. For
each parallel pair, the student is trained so that both the source sentence and
its translation land on the teacher's vector for the source, using a squared
error [arxiv:2004.09813 §2] [arxiv:2004.09813 §3]. The authors use XLM-R as the
student, because its vocabulary covers many languages [arxiv:2004.09813 §3].
Card: [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813).

The paraphrase-trained teacher gives the best numbers. Its student averages
84.6 on monolingual STS 2017 and 83.7 on cross-lingual STS 2017, against 73.5
for LaBSE and 67.0 for LASER on the cross-lingual set [arxiv:2004.09813 tab. 1]
[arxiv:2004.09813 tab. 2]. On Tatoeba search over four low-resource languages,
the student scores above LASER in both directions [arxiv:2004.09813 tab. 4].
The authors report these numbers themselves.

### Arabic encoders on a per-cluster benchmark suite (ARBERT, MARBERT and ARLUE)

Pre-train a BERT-Base encoder on one kind of Arabic text, then fine-tune it on
each task in a shared suite and report one score. ARBERT is pre-trained on
formal Arabic and MARBERT on tweets, which include dialects [arxiv:2101.01785 §3.1]
[arxiv:2101.01785 §3.2]. The suite, ARLUE, has 42 datasets. Each fine-tuning run
uses Adam, batch 32 and learning rate 2e-6 for 25 epochs, with the best epoch
chosen on the dev split [arxiv:2101.01785 §3.4]. Card:
[ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785).

The ARLUE score is the average over eight table rows, and MARBERT-v2 gets 77.40,
against 76.55 for XLM-R Large and 76.07 for ARBERT [arxiv:2101.01785 §5.2].
By row, MARBERT leads on dialect region and country, while ARBERT leads on NER
[arxiv:2101.01785 §5.2]. These are author-reported single runs.

### Zero-shot chat model across seven tasks (ChatGPT, Lai et al.)

Send one prompt per example to a chat model. The prompt is a short task
description, then the input, and the reply is mapped to the task's labels by
rules the paper does not publish [arxiv:2304.05613 §3]. The authors cover POS
tagging, NER, relation extraction, NLI, QA, commonsense reasoning and
summarization [arxiv:2304.05613 §4]-[arxiv:2304.05613 §10]. Prompt format
varies by task: POS and NER take a list of words, NER asks for BIO labels, and
summarization adds an output-language line [arxiv:2304.05613 §4] [arxiv:2304.05613 §5]
[arxiv:2304.05613 §10]. Each language is placed in a resource class by its
share of CommonCrawl: above 1% is high, above 0.1% medium, above 0.01% low,
and the rest extremely low [arxiv:2304.05613 §3]. Card:
[ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](/papers/2304.05613).

The authors give both English and target-language task descriptions for
most tasks [arxiv:2304.05613 §3]. Averages over the languages in each task
show the same pattern. POS tagging is the exception: ChatGPT with English
descriptions averages 84.5 against 79.3 for XLM-R [arxiv:2304.05613 §4]. On
the other six tasks the supervised baselines lead on average. NLI is 57.0
against 87.1 for mT5-XXL, and QA F1 is 53.5 against 85.2 [arxiv:2304.05613 §7]
[arxiv:2304.05613 §8]. NER macro-F1 is 29.9 against 88.4 for DAMO
[arxiv:2304.05613 §5]. Relation extraction micro-F1 is 69.4 against 85.0 for
mT5-IL [arxiv:2304.05613 §6], and commonsense accuracy is 47.8 against 59.0
for TRT [arxiv:2304.05613 §9]. The authors report English descriptions as
usually better [arxiv:2304.05613 §11]. These are author-reported averages over
each task's language set.

### Human audit of language subsets in web-crawled corpora (Kreutzer et al.)

Draw a random sample of up to 100 lines for each language subset, then have a
speaker, or a rater with a dictionary and web search, label each line with a
fixed taxonomy. The labels separate correct text, correct but boilerplate text,
text in the wrong language, and non-linguistic content. Shares are then
reported per language, macro (each language equal) and micro (weighted by
size) [arxiv:2103.12028 §4.1] [arxiv:2103.12028 §4.2] [arxiv:2103.12028 tab. 3].
Card: [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](/papers/2103.12028).

The audit covers 205 subsets of five corpora. Macro correct-share ranges from
23.74% (WikiMatrix) to 87.21% (OSCAR), and 15 subsets contain no correct
sentence [arxiv:2103.12028 tab. 3] [arxiv:2103.12028 §4.2]. Labels come from one
rater per line, so the authors call the per-language numbers rough
[arxiv:2103.12028 §4.1]. The paper also tests automatic filters. CLD3 reaches
40.6% average precision on CCAligned sentence pairs, and a Transformer LangID
filter raises median precision from 13.8% to 43.9% while losing 77.5% of recall
[arxiv:2103.12028 §4.3].

### Hofstede survey items in three prompt types (Cao et al.)

Take 24 questions from the Hofstede Culture Survey and change them to third
person. Prefix each with a culture phrase, and ask them in three ways: in
English with a nationality, in the country's own language, and in English with
"In the [country] culture setting" [arxiv:2303.17466 §3.1] [arxiv:2303.17466 tab. 1].
Re-ask until the model names an option, then map the answer to a 1 to 5 score.
Six dimension scores come from a weighted difference of question pairs
[arxiv:2303.17466 eq. 1] [arxiv:2303.17466 tab. 9]. Each culture's six scores
are then correlated with human scores using Spearman's rank correlation
[arxiv:2303.17466 §4]. Card: [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](/papers/2303.17466).

The probe uses five cultures: US, China, Germany, Japan and Spain. Prompts 1
and 3 are both English, and they agree on over 70% of questions in every
culture except China [arxiv:2303.17466 tab. 3] [arxiv:2303.17466 §4.1]. Only one Spearman cell
reaches p < 0.05, Spain with prompts 1 and 2 at 0.93 [arxiv:2303.17466 tab. 5].
The authors read the results as best alignment with the US [arxiv:2303.17466 §4.2].
The paper gives no human reference values in its text and no repeat count
[arxiv:2303.17466 app. A.1] [arxiv:2303.17466 §4].

### Translate-then-classify for depression severity (Skianis et al.)

Translate each English Reddit post into Greek with GPT-3.5-turbo at temperature
0. Then classify both the English original and the Greek translation with one
zero-shot prompt that asks for one of four severity levels, coded 0 to 3
[arxiv:2410.12985 §3] [arxiv:2410.12985 tab. 1]. A reply with no recognised
label counts as class 0 [arxiv:2410.12985 §5]. The English run is the only
baseline [arxiv:2410.12985 tab. 2]. Card: [Leveraging LLMs for Translating and Classifying Mental Health Data](/papers/2410.12985).

On DepSeverity's 3,553 posts, the macro F1 is 0.17 in English and 0.16 in
Greek [arxiv:2410.12985 tab. 2]. The Minimum class holds about 73% of posts,
and the Minimum recall is 0.14 in English [arxiv:2410.12985 tab. 1]
[arxiv:2410.12985 tab. 2]. The paper does not measure translation quality
[arxiv:2410.12985 §4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Only an LLM behind an API, no training budget | Five-shot or zero-shot English prompts | No training is needed. Five-shot base models and zero-shot chat models are reported separately, so say which you ran [arxiv:2308.16884 tab. 2] |
| Language model that is strong in English but weak in the target language | Translate-test, with the translation system named | The paper finds translate-test ahead of in-language for most languages it tested. Only German and Italian were better in-language, and none of the languages where translate-test won is high-resource [arxiv:2308.16884 §5.2] [arxiv:2308.16884 tab. 3]. The gain depends on translation quality, which the paper does not measure |
| Masked multilingual encoder, a training budget, and a non-commercial data source is acceptable | Translate-Train-All | Highest encoder averages in the paper [arxiv:2308.16884 tab. 2] |
| One target language, a corpus of billions of words for it, and a training budget | A monolingual encoder, scored on a per-language suite such as GLUES | The authors report it ahead of the best multilingual BERT on most tasks, but not on every QA setting [arxiv:2308.02976 §5.2] [arxiv:2308.02976 tab. 2]. Check the corpus licences before commercial use (see Pitfalls) |
| Commercial product or model that must be shipped | Fine-tuning on the Belebele English training set is not allowed | The training set is largely CC BY-NC [arxiv:2308.16884 app. A.3]. The paper says a different training source is needed. The test set is CC BY-SA [arxiv:2308.16884 app. A.3] |
| Language written in two scripts (native and Latin) | Score both scripts | Native script beat Latin-script romanization for all models except Falcon, on six languages [arxiv:2308.16884 §5.2] [arxiv:2308.16884 app. A.7.3] |
| Need to see how the model handles the long tail, not just the average | Report the share of languages above 50% and 70% | An average can hide weak languages. For LLaMA 2 70B five-shot, 38.5% of variants reach 50 and 26.2% reach 70 [arxiv:2308.16884 tab. 2] |
| Many tasks in many languages, with a generative model and few labels | Label-likelihood scoring with the same templates for every language | This is the mGPT protocol. Use it for trends across tasks, and compare only against baselines run the same way, since its fine-tuned baselines are not like-for-like [arxiv:2204.07580 §4.2] [arxiv:2204.07580 tab. 9] |
| Choosing which languages a benchmark must cover, or checking how far a research community covers them | Count languages per resource class, from labelled and Wikipedia data, and report the share in each class | The paper finds 549 of 1,139 WALS categories occur only in the three lowest classes [arxiv:2004.09095 §3]. Its class cut-offs are not stated, so set your own thresholds and say what they are [arxiv:2004.09095 §2.3] |
| Checking whether a model leans toward one culture's entities in Arabic | Cultural Bias Score on masked prompts, using the table averages | It needs token probabilities, so closed APIs that do not return them cannot be scored. GPT-4 has no score for this reason [arxiv:2305.14456 app. D] [arxiv:2305.14456 §4.4] |
| Checking survey-style values in a target country | Persona prompts in the dominant language, with both hard and soft scores | The authors find GPT-3.5 and AceGPT do best in the country's dominant language on both scores [arxiv:2402.13231 §5.2] [arxiv:2402.13231 tab. 2]. Report both scores, since they can disagree |
| Trying to raise alignment for under-represented personas | Anthropological prompting, tested first on your own model and survey | The gain was measured on one model and one survey, and the paper's claims about underrepresented groups rest on a figure [arxiv:2402.13231 §5.6] [arxiv:2402.13231 fig. 5] |
| Web-crawled corpus in many languages, before training on it or publishing it | Read about 100 lines per language subset with a fixed taxonomy, and check the language codes. Do not rely on an automatic LangID filter alone | The correct share rises with corpus size in every corpus (Spearman 0.37 to 0.66), so the smallest subsets are the ones to check first. CLD3 precision was 40.6% on CCAligned pairs [arxiv:2103.12028 §7] [arxiv:2103.12028 §4.2] [arxiv:2103.12028 §4.3] [arxiv:2103.12028 §5] |
| Measuring whether a chat model's stated values differ by prompt language | Survey items in each prompt type, with consistency across prompts reported beside any correlation | Agreement between prompt types ranges from 37.50% to 79.17% in the paper's table, and only one of the 18 reported Spearman cells reaches p < 0.05 [arxiv:2303.17466 tab. 3] [arxiv:2303.17466 tab. 5] |
| Classifying a language with no labelled data by first translating with the same LLM | Do not deploy on this evidence. If you test it, score translation and classification separately, with a human check on both | The paper reports macro F1 of 0.16 to 0.17 in both languages and measures no translation quality. It says the approach is not meant to diagnose or assist patients [arxiv:2410.12985 tab. 2] [arxiv:2410.12985 §4] [arxiv:2410.12985 §5] |
| Code model that must work in several programming languages | Unit-test pass@k on a set translated by hand into each language | Scores by running tests, where earlier multilingual code sets scored by string similarity such as BLEU [arxiv:2303.17568 §1] [arxiv:2303.17568 §3.1]. pass@k needs many samples per problem, so budget the sampling cost [arxiv:2303.17568 §3.2] |
| Word-similarity test for a language with no published list, and native speakers to translate and rate pairs | Translate and re-rate an English list, as in this paper, with an agreement check | Yields a per-language test set with reported agreement, Fleiss' kappa from 0.756 (Tamil) to 0.887 (Urdu) [arxiv:2609.34138 tab. 1]. The lists are small (97 to 163 pairs), so a few pairs can move the score [arxiv:2609.34138 §2] [arxiv:2609.34138 §6] |
| Multilingual sentence similarity, with an English sentence model you trust and parallel text for the target languages | Distillation into XLM-R, scored on cross-lingual similarity | The paper's student beats LASER, mUSE and LaBSE on cross-lingual STS 2017 [arxiv:2004.09813 tab. 2]. It trails LASER and LaBSE on BUCC bitext mining, so it suits semantic similarity more than exact translation pairs [arxiv:2004.09813 tab. 3] [arxiv:2004.09813 §4.2] |
| Exact translation pairs, such as bitext mining | LASER or LaBSE, not the distilled student | The BUCC F1 scores favour them over the distilled student [arxiv:2004.09813 tab. 3] |
| Arabic encoder on a mix of formal text and dialect, with a per-task fine-tune | ARBERT for formal text, MARBERT for tweets and dialect | The authors report MARBERT ahead on dialect rows and ARBERT ahead on NER [arxiv:2101.01785 §5.2]. The checkpoints are for research use, and commercial use needs the authors' agreement [code: README.md#L262-L263], so check before shipping |
| Quick zero-shot read of a hosted chat model across several tasks, with no training budget | Per-task prompts in English, with a supervised baseline for each task | The paper finds the chat model far behind supervised systems on most tasks, with POS tagging the exception [arxiv:2304.05613 §4] [arxiv:2304.05613 §11]. The model is not pinned, so record the date and endpoint with the numbers (see Pitfalls) |
| Checking whether a model matches one country's survey answers and school knowledge | KorNAT-style scoring, with the harness letter order and the leaderboard scorer | Uses the survey's answer distribution as the gold label, not one answer [arxiv:2402.13605 §3.1.3] [arxiv:2402.13605 §4]. Scoring runs on a server the harness does not contain, so check your numbers against it (see Pitfalls) |

## Implementation sketch

Score every language variant with the same items, then report per-language
accuracy and a summary. The common core is the same for all four variants. The
variants differ in how the prompt is built and whether the model is trained
first.

```pseudocode
# Score a parallel benchmark across language variants  [arxiv:2308.16884 §3]
# Input:  variants, map language -> list of items with a shared item id
# Output: accuracy per language, and the share of languages above 50

def score_parallel(model, variants, build_prompt):
    # Check that every variant holds the same item ids  [arxiv:2308.16884 §3]
    ids = None
    for lang, items in variants.items():
        lang_ids = {item.id for item in items}
        if ids is None:
            ids = lang_ids
        if lang_ids != ids:
            raise ValueError("variant is not parallel: " + lang)
    # Score each language with the same prompt builder  [arxiv:2308.16884 §4.2]
    accuracy = {}
    for lang, items in variants.items():
        correct = 0
        for item in items:
            # Choose one letter from A to D  [arxiv:2308.16884 app. A.4.2]
            prompt = build_prompt(item, lang)
            letter = model.best_letter(prompt, "ABCD")
            if letter == item.answer:
                correct += 1
        accuracy[lang] = correct / len(items)
    # Report the share of languages above 50 and 70  [arxiv:2308.16884 tab. 2]
    share_50 = sum(a > 0.5 for a in accuracy.values()) / len(accuracy)
    return accuracy, share_50
```

For translate-test, `build_prompt` first translates the passage, question and
options into English [arxiv:2308.16884 §4.2]. For fine-tuning, `model` is the
trained classifier and `best_letter` scores the four options
[arxiv:2308.16884 app. A.4.1]. For the Cultural Bias Score, replace the
accuracy with a count of pairs where the Western entity wins at the mask
[arxiv:2305.14456 §4.4].

## Contradictions

- **Translating the input.** Belebele finds that translate-test beats
  in-language scoring for most languages it tests, on LLaMA 2-chat 70B
  [arxiv:2308.16884 tab. 3]. CAMeL argues that translated English datasets lose
  the cultural identity of the test data, so translation is not an effective
  route for cultural evaluation [arxiv:2305.14456 app. A]. The two papers
  measure different things: accuracy on a reading task, against cultural bias
  or fit. The translation step is also different in each. The reconciliation
  is not stated in either paper.
- **Native-language prompting.** AlKhamissi et al. find that GPT-3.5 and
  AceGPT-Chat fit each survey better in its dominant language
  [arxiv:2402.13231 §5.2]. This is a within-language comparison, not a
  translate-test one, so it does not directly test the Belebele result.
  Cao et al. also say most cultures align better when probed in their own
  language [arxiv:2303.17466 §4.2], and the native-language rows of their table
  are higher for China, Germany, Japan and Spain [arxiv:2303.17466 tab. 5]. Their
  Spearman cells are mostly not significant, so this agrees only in direction.

Inside the Belebele paper, the text says seven languages appear in two scripts,
while §3.3 names five Indo-Aryan languages and Modern Standard Arabic, which is
six [arxiv:2308.16884 §1] [arxiv:2308.16884 §3.3]. The language table is the
reference [arxiv:2308.16884 tab. 4].

## Pitfalls

- **Resource classes are not reproducible.** The class cut-offs are not given,
  the language-mention detector is unspecified, and the dataset is not yet
  released. No code is linked, so the paper's tables cannot be rebuilt from its
  text [arxiv:2004.09095 §2.3] [arxiv:2004.09095 app. A.2].
- **Baselines are literature values (BETO).** The multilingual rows in its
  tables are the best published results per task. Dev or test is not stated,
  and no variance is reported, so gaps of a few tenths of a point need care
  [arxiv:2308.02976 tab. 1] [arxiv:2308.02976 §5.2].
- **Corpus licences (BETO).** The Spanish pre-training corpus is drawn from OPUS
  sources, including subtitles and news. The authors say the licences of the
  training datasets may not be compatible with CC BY 4.0 for commercial use, so
  check each source before using the weights or the corpus [arxiv:2308.02976 §3]
  [code: README.md#L58-L59].
- **Test set, not training set.** Do not tune or train on Belebele test items
  [arxiv:2308.16884 §3.4].
- **Training set licence.** The English training set is assembled from six
  datasets, the majority under CC BY-NC [arxiv:2308.16884 app. A.3]. Do not use
  it for commercial fine-tuning.
- **Chance level depends on the protocol.** The 0.25 chance level applies to
  classifiers. Generation with exact-match scoring has no such floor
  [arxiv:2308.16884 §4].
- **Prompt formats differ.** The five-shot template uses P, Q and Answer labels.
  The zero-shot prompt uses a different layout [arxiv:2308.16884 app. A.4.2]
  [arxiv:2308.16884 app. A.4.3]. Report which one you used.
- **Lenient answer parsing.** The released sample code takes the first
  character of an unrecognised output as the answer. The paper does not describe
  this fallback [arxiv:2308.16884 app. A.4.3]
  [code: sample_zero_shot_instructions.md#L40-L45]. Report parse-failure rates.
- **Learning rate printed as a range.** The XLM-V Translate-Train-All learning
  rate is printed as "3-6". The paper does not state the intended value
  [arxiv:2308.16884 tab. 6].
- **Translation quality.** Literal translations can make the same question
  harder in some languages. Annotators found quality problems in the FLORES
  translations, and the authors changed questions to fit the translated
  passages instead of correcting FLORES [arxiv:2308.16884 Limitations].
- **Human baseline is a sample.** The 97.6 human score comes from four authors
  answering about 30 English items each. The 95% interval covers all 900 items
  [arxiv:2308.16884 §5.1].
- **Closed models cannot be audited.** GPT-3.5 Turbo's training data is not
  disclosed, and the paper warns that comparisons with it may be unfair
  [arxiv:2308.16884 Limitations] [arxiv:2308.16884 §4.1].
- **Averages hide English.** LLaMA 2 70B reaches 90.9 in English but averages
  48.0 across variants [arxiv:2308.16884 tab. 2]. Report English next to the
  average.
- **Abstract wording versus tables (mGPT).** The summary says the 13B model is
  on par with XGLM. Its few-shot results are below XGLM-7.5B on most tasks, so
  use the table [arxiv:2204.07580 tab. 7].
- **Perplexity normalisation conflicts (mGPT).** Equation 1 normalises by
  characters, but the language-modelling section normalises by tokens. Compare
  perplexities only within one convention [arxiv:2204.07580 eq. 1]
  [arxiv:2204.07580 §3.2] [arxiv:2204.07580 §4.1].
- **Chinese is evaluated but not in the 61 languages (mGPT).** The code runs
  Chinese on several tasks, though the paper's language list comments it out.
  State the language set you report on [arxiv:2204.07580 tab. 1].
- **CBS ranges disagree (CAMeL).** The text says 40 to 60%, a figure caption
  says 40 to 65%, and the table caption says 30 to 80%. The table averages run
  from 45.5 to 63.5, so use those [arxiv:2305.14456 §4.4] [arxiv:2305.14456 tab. 10].
- **Ties and resampling (CAMeL).** Ties count as zero, and each run draws new
  entities, so a score varies with the sample. Report the standard deviation
  over runs [arxiv:2305.14456 §4.4] [arxiv:2305.14456 tab. 10].
- **Hosted models change (CAMeL).** GPT-3.5 is text-davinci-003 and GPT-4 is
  gpt-4-1106-preview. Results on these endpoints cannot be reproduced once they
  are retired [arxiv:2305.14456 app. D].
- **One vote or four paraphrases (survey).** The paper majority-votes each
  paraphrase, but its formula reads as one vote per question and persona
  [arxiv:2402.13231 §4.5] [arxiv:2402.13231 eq. 1]. Decide which you implement
  and say so.
- **Invalid answers are dropped (survey).** Rows with no valid parsed option
  are removed, not scored as wrong, so coverage can differ between models. The
  paper does not report invalid-answer rates [code: scripts/utils.py#L360-L364].
- **Units differ between tables (survey).** Tables 2 and 3 are percentages, and
  table 4 is fractions. The vanilla row in table 4 does not match the Egypt
  English row in table 2 [arxiv:2402.13231 tab. 2] [arxiv:2402.13231 tab. 4].
- **"Significantly" without a test (survey).** The paper calls the US gap
  significant but reports no statistical test. Treat the gaps as descriptive
  [arxiv:2402.13231 §5.1].
- **Licences of code and data.** The mGPT repository's code licence is unclear.
  The AlKhamissi repository has no LICENSE file. The CAMeL prompts are derived
  from social-media posts and the repository is MIT. Describe these methods in
  your own words and check each data licence before reuse [arxiv:2204.07580 §3.1]
  [arxiv:2402.13231 §4.1] [arxiv:2305.14456 Ethics Statement].
- **Code weights are non-commercial (CodeGeeX).** The model licence allows
  non-commercial research only and is governed by Chinese law. The repository's
  Apache-2.0 code licence does not extend to the weights, and the paper's CC BY
  licence covers the paper, not the model [code: MODEL_LICENSE §2-§3, §6].
  Do not use these weights in a product.
- **pass@k is reported only with enough samples (CodeGeeX).** The released
  code reports pass@k only when every problem has at least k samples
  [code: codegeex/benchmark/evaluate_humaneval_x.py#L209-L211]. Fewer samples
  silently change which k values appear.
- **Temperature for pass@10 conflicts (CodeGeeX).** Section 4.1 uses 0.8, and
  the table caption gives 0.6 [arxiv:2303.17568 §4.1] [arxiv:2303.17568 app. A.4.1].
  Use 0.8 unless you are reproducing the caption.
- **No reference score for social values (KorNAT).** The introduction says two
  of seven models exceed "our reference score" on social values, but no
  threshold is stated. Only All-Neutral and the maximum rows are defined
  baselines [arxiv:2402.13605 §1] [arxiv:2402.13605 tab. 1].
- **Letter order is reversed in the harness (KorNAT).** The paper lists the
  survey scale from Strongly Disagree to Strongly Agree. In the harness, A is
  Strongly Agree. Use the harness order when producing submissions
  [code: kornat_eval.py#L59-L63] [arxiv:2402.13605 §3.1.3].
- **Item counts differ (KorNAT).** The paper reports 6,000 common-knowledge
  items, while the harness expects 10,000 [arxiv:2402.13605 app. B.2]
  [code: kornat_eval.py#L50].
- **Some cross-national rows look duplicated (KorNAT).** The USA prompt rows for
  HyperCLOVA X and Gemini Pro match in almost every cell, and the paper does not
  explain it. The text also says both prompts help every model except Claude-1,
  but Gemini Pro falls below its baseline on A-SVA and N-SVA in the table
  [arxiv:2402.13605 tab. 2] [arxiv:2402.13605 §5.2.2].
- **Refusals are dropped, denominators are not stated (KorNAT).** The paper
  omits refused and invalid responses, but does not say whether they count as
  wrong [arxiv:2402.13605 §5.4]. The harness scores a social refusal as 0
  [code: README.md#L25-L26].
- **Pair selection is not stated (word similarity).** The paper names RG-65 and
  WordSim-353 as the bases but does not say how 97 to 163 pairs were chosen
  from them [arxiv:2609.34138 §4.1]. The sampling cannot be reproduced from the
  text.
- **Zero vectors (word similarity).** Missing words get a zero vector, and
  cosine with a zero vector is undefined. The paper does not say what score
  that gives, and it is unclear whether the rule applies to the "FastText w/
  OOV" row, which generates vectors [arxiv:2609.34138 §2] [arxiv:2609.34138 §6].
- **Agreement scale (word similarity).** Ratings are on a 0 to 10 scale and
  averaged, but agreement is reported as Fleiss' kappa. The paper does not say
  how the scale became categories for kappa [arxiv:2609.34138 §4.2]
  [arxiv:2609.34138 tab. 1].
- **Multiple choice only (KorNAT).** Free-form generation was not evaluated, so
  the scores do not show open-ended behaviour [arxiv:2402.13605 §7].
- **Script teacher differs from the paper (distillation).** The paper's teacher
  is an NLI and STS model, but the example script uses a paraphrase model and
  five epochs, not 20 [arxiv:2004.09813 §3] [code: make_multilingual.py#L45]
  [code: make_multilingual.py#L54]. Choose one setup and report it.
- **Loss scale differs from the paper (distillation).** The script's loss is
  smaller than the paper's sum by a factor tied to the embedding dimension, so
  a fixed loss threshold will not transfer [arxiv:2004.09813 §2]
  [code: sentence_transformers/sentence_transformer/losses/embed_distill.py#L214-L225].
- **BUCC labels are incomplete (distillation).** The authors inspected 60 false
  positives and found 57 were valid translations that the dataset did not label
  as parallel [arxiv:2004.09813 §4.2]. Read BUCC scores with care.
- **Tatoeba may flatter the student (distillation).** LASER was trained on
  Tatoeba, and the distilled student was not, so the comparison may favour the
  student [arxiv:2004.09813 app. A].
- **The ARLUE score is not a six-cluster average (Arabic).** The abstract speaks
  of six clusters, but the score is the average over eight table rows, because
  dialect is split three ways. Averaging the eight rows, each over its two
  metrics, reproduces the reported 77.40 [arxiv:2101.01785 §5.2].
- **MARBERT-v2 is never defined (Arabic).** The results table reports it, but the
  paper does not say how it differs from MARBERT. Treat the name as a label only
  [arxiv:2101.01785 §5.2].
- **Arabic checkpoints are for research (Arabic).** Do not ship them in a
  commercial product on the strength of this concept page
  [code: README.md#L262-L263].
- **Model and settings not pinned (ChatGPT).** The paper names "ChatGPT" and
  the GPT-3.5 series, with replies collected from 1 March to 5 April 2023.
  Sampling settings, max tokens and the snapshot are not stated, and the
  interface (web or API) is not named [arxiv:2304.05613 §1] [arxiv:2304.05613 §3].
  Hosted models change, so these numbers cannot be rerun exactly.
- **Parse rules are not published (ChatGPT).** The paper's formula nests the
  reply step inside the parser, and the parse rules are described only as
  task-specific. Parse failures cannot be separated from model errors
  [arxiv:2304.05613 §3].
- **Language count differs (ChatGPT).** The method says 34 languages and the
  introduction and conclusion say 37 [arxiv:2304.05613 §3] [arxiv:2304.05613 §1].
  Use the per-task tables, not the headline count.
- **Corpus audit labels come from one rater (web corpora).** Each sentence has
  one label, and some raters had no speaker of the language. The authors put
  accuracy against expert labels at 0.66 on six classes for CCAligned, 0.79 on
  two classes [arxiv:2103.12028 tab. 4] [arxiv:2103.12028 §4.1]. Dialects were
  not handled [arxiv:2103.12028 §7].
- **Macro and micro give different answers (web corpora).** CCAligned is 29%
  correct macro but 54% micro, and OSCAR is 87% macro and 99% micro
  [arxiv:2103.12028 tab. 3]. Say which one you quote.
- **Language codes must be checked (web corpora).** The paper finds 83
  nonstandard codes, and at least 48 of the 83 affected corpora are entirely
  spurious. It names deprecated codes such as `sh` and `iw`
  [arxiv:2103.12028 §5] [arxiv:2103.12028 app. A].
- **Repeat count and sample size are missing (Hofstede probe).** The number of
  re-asks, the decoding settings and the model version are not stated. The
  Spearman sample is not stated either. Six dimensions per culture would give
  n = 6, which is an inference that the p-values depend on [arxiv:2303.17466 §4]
  [arxiv:2303.17466 tab. 5].
- **Table 4 rows for uai and mas look swapped (Hofstede probe).** Applying eq. 1
  with the paper's coefficients to the Prompt 1 scores gives, for uai, the
  values printed in the mas row, and the reverse for mas. Check the rows before
  you rely on them [arxiv:2303.17466 tab. 4] [arxiv:2303.17466 tab. 9]
  [arxiv:2303.17466 tab. 10].
- **Scale direction varies (Hofstede probe).** Some questions list "always"
  first and others "never" first, and the paper does not say whether any scores
  were reversed [arxiv:2303.17466 app. A.1]. Use the printed option numbers.
- **Free text is mapped to options without a rule (Hofstede probe).** Half-point
  scores come from averaging several named options, and no inter-rater check is
  reported [arxiv:2303.17466 tab. 10].
- **No scoring code for the Hofstede probe.** The repository the paper links
  holds data files and a README, with no scoring script and no licence, so the
  scores cannot be checked against code [arxiv:2303.17466 app. A.5]. Re-derive
  them from the paper's tables.
- **Greek prompt is not stated (translate-then-classify).** The paper does not
  say whether the classification prompt was also translated for the Greek run.
  Choose one and report it [arxiv:2410.12985 §4] [arxiv:2410.12985 tab. 1].
- **Translation quality is not measured (translate-then-classify).** The only
  check is a qualitative look at two examples [arxiv:2410.12985 §4]. The Greek
  scores mix classifier and translation effects.
- **Minimum class dominates (translate-then-classify).** The Minimum class has
  2,587 of 3,553 posts. Its recall is 0.14 in English, so the model rarely
  predicts it, and the macro average weights the rare classes equally
  [arxiv:2410.12985 tab. 1] [arxiv:2410.12985 tab. 2]. Report per-class scores
  with the macro.
- **Macro row is rounded differently (translate-then-classify).** The mean of
  the four English F1 values is about 0.178, and the table prints 0.17. The
  paper does not say how the macro row was computed [arxiv:2410.12985 tab. 2].
- **Low-resource replies often fail (ChatGPT).** For Thai, Bengali and
  Burmese, only 42 to 59 percent of summarization requests returned a non-empty
  reply with English descriptions, so some low scores reflect failed requests
  [arxiv:2304.05613 §10].

## Papers

- [Joshi et al., language resource classes in NLP venues](/papers/2004.09095): six language resource classes, and how inclusive NLP venues are toward each class.
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813): an English sentence encoder distilled into a multilingual student, scored on cross-lingual STS, BUCC and Tatoeba, with author-reported comparisons to LASER, mUSE and LaBSE.
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](/papers/2101.01785): two Arabic BERT-Base encoders and the 42-dataset ARLUE suite, with author-reported scores.
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](/papers/2308.16884): a 900-question parallel reading set in 122 variants, with four protocols and author-reported results for each.
- [mGPT: Few-Shot Learners Go Multilingual](/papers/2204.07580): two decoder-only models in 61 languages, scored by label likelihood across classification, NER, POS and probing.
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](/papers/2305.14456): the CAMeL Arabic prompt set and the Cultural Bias Score for Arab versus Western entity preference.
- [Investigating Cultural Alignment of Large Language Models](/papers/2402.13231): persona-based simulation of the World Values Survey in Egypt and the US, with hard and soft scores.
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](/papers/2304.05613): zero-shot ChatGPT on seven NLP tasks in 34 to 37 languages, with author-reported comparisons to supervised baselines.
- [Spanish Pre-trained BERT Model and Evaluation Data](/papers/2308.02976): a Spanish-only BERT-Base encoder and GLUES, a Spanish task collection, with author-reported comparisons to multilingual BERT.
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](/papers/2303.17568): a 13B code model and HumanEval-X, a five-language unit-test benchmark, with author-reported pass@k.
- [Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems](/papers/2609.34138): six new word-similarity sets built by translation and native-speaker rating, with CBOW, Skip-gram and FastText baselines for three languages.
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](/papers/2402.13605): a Korean benchmark that scores social-value answers against a national survey and common knowledge against school curricula.
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](/papers/2103.12028): a human audit of 205 language subsets from five web-crawled corpora, with an error taxonomy and tests of automatic LangID filters.
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](/papers/2303.17466): Hofstede survey items asked in three prompt types for five cultures, with author-reported Spearman correlations to human scores.
- [Leveraging LLMs for Translating and Classifying Mental Health Data](/papers/2410.12985): GPT-3.5-turbo translates Reddit posts into Greek and classifies depression severity, with author-reported macro F1 in English and Greek.