Multilingual evaluation

Overview

Multilingual evaluation asks how well a model performs in each language it is meant to serve, and whether the numbers can be compared. The invariant that makes comparison possible is parallel data: the same items exist in every language, so a score gap comes from the language or the model, not from harder questions 2308.16884 §3. The authors note that most multilingual reading-comprehension benchmarks cover around 30 languages 2308.16884 §1.

Belebele is the source here. It has 900 four-choice reading questions on 488 passages, in 122 language variants, so each language has 900 rows and the dataset has 109,800 rows in total 2308.16884 §3.5 2308.16884 tab. 1. Questions are written in English first. Translators then render each passage and align its questions to the translated text, even where the translation is ambiguous 2308.16884 §3.3. The passages come from FLORES-200, which native speakers translated 2308.16884 §2.4 2308.16884 §3.3.

Multiple choice was chosen over span extraction. Span extraction is sensitive to morphology and is harder to scale across languages 2308.16884 §3.1. The benchmark is a test set. The authors say it should not be used for training 2308.16884 §3.4.

Parallel items are not the only design in the cards. Other designs measure different things, and their numbers should not be compared directly:

• Label-likelihood suites score a generative model on many tasks in many languages, with a few demonstrations 2204.07580 §4.2.
• Cultural-preference probes check whether a model prefers culturally Western or Arab entities in Arabic text 2305.14456 §4.4.
• Survey simulation asks a model to answer as a survey respondent in a given country and language 2402.13231 §4.5.
• Single-language suites test a model pre-trained on one language against the best multilingual baseline the authors could find, on tasks in that language 2308.02976 §4 2308.02976 tab. 1.
• Resource-class audits score no model. They sort languages into six classes by labelled and Wikipedia data, then measure how far NLP venues mention each class 2004.09095 §2.2 2004.09095 §2.3.
• Country-specific alignment scores a model against one country's survey answers and school-curriculum questions 2402.13605 §3 2402.13605 §4.
• Multilingual code generation runs one set of programming problems in several programming languages and scores each by unit tests 2303.17568 §3.1 2303.17568 §3.2. Here "multilingual" means programming languages, not natural languages, so these numbers do not transfer to the other designs.
• Human-rated word-similarity sets add a test list for a language that has none, then score word vectors by rank correlation with human ratings. Each list is monolingual and is built by translating two English lists, so its scores are per language 2609.34138 §4.1 2609.34138 §2.
• Distilled sentence encoders train a multilingual student to match a fixed English teacher's vectors on translation pairs, then score cross-lingual similarity and bitext mining 2004.09813 §2 2004.09813 §4.1.
• Single-language benchmark suites test one language across many tasks at once, here Arabic over 42 datasets 2101.01785 §5.1.
• Zero-shot chat model on existing task sets sends one prompt per example to a chat model, with no demonstrations, across seven NLP tasks in 34 to 37 languages. Each task uses its own existing dataset, not a shared parallel set 2304.05613 §3 2304.05613 §1.
• Human audit of web-crawled corpora reads random lines from a multilingual training or release corpus and labels each one, with no model involved. It checks whether a subset contains the language its code claims 2103.12028 §4.1 2103.12028 §4.2.
• Survey probe across prompt languages asks a chat model the same Hofstede culture-survey items in English with a country named, in the country's language, and in a second English wording 2303.17466 §3.1.
• Translate-then-classify has one LLM translate English posts into Greek, then classify the translations for depression severity. It is scored against the labels of the English originals 2410.12985 §3 2410.12985 §4.

Variants

The first five variants are protocols for scoring the same parallel benchmark (Belebele). Each is a protocol, not a different dataset 2308.16884 §4.2. The remaining variants use other designs.

Fine-tune a masked encoder on English (zero-shot transfer)

Fine-tune a masked multilingual model with a multiple-choice head on the English training set, then test it in every language without further training. This is zero-shot cross-lingual transfer 2308.16884 §4.2 2308.16884 app. A.4.1. Card: The Belebele Benchmark. Full fine-tuning of XLM-R, XLM-V and InfoXLM (large) averages 54.0 to 56.2 across 122 variants 2308.16884 tab. 2.

Fine-tune on machine-translated training data (Translate-Train-All)

Machine-translate the English training set into every target language and fine-tune on the combined data, for one epoch with a cap of 650k samples 2308.16884 app. A.4.1. This variant scores highest among the encoder results: 58.9 to 60.2 average 2308.16884 tab. 2. The paper does not name the machine-translation system 2308.16884 §4.2. Card: The Belebele Benchmark.

Few-shot prompting in English (five-shot in-context learning)

Prepend five English examples from the training set to each test item. The answer is the letter with the highest probability among A to D 2308.16884 §4.2 2308.16884 app. A.4.2. The paper averages three runs and does not state seeds 2308.16884 app. A.4.2. Card: The Belebele Benchmark.

Zero-shot English instructions

Give an English instruction, the target-language item, and read the answer letter from the generated output 2308.16884 §4.2 2308.16884 app. A.4.3. Card: The Belebele Benchmark.

Translate-test

Machine-translate the passage, question and options into English, then use an English prompt 2308.16884 §4.2 2308.16884 §5.1. For LLaMA 2-chat 70B, translate-test averages 57.1 over 91 non-English languages, against 44.1 for the same items in-language 2308.16884 tab. 3. Card: The Belebele Benchmark.

Label likelihood with few-shot demonstrations (mGPT suite)

Score each candidate label, or each candidate tag for one word, by the summed negative log-likelihood of the full prompt. Demonstrations come before the example 2204.07580 §4.2.1. For sequence labelling, the model tags words left to right, and each word's tag is chosen the same way 2204.07580 §4.2.2. The suite covers classification (XWINO, PAWSX, XCOPA, XNLI and a hate-speech set), NER and POS, and knowledge probing on mLAMA in 23 languages 2204.07580 §4.2 2204.07580 §4.3. Card: mGPT: Few-Shot Learners Go Multilingual.

The authors compare two decoder-only models, about 1.3B and 13B parameters, with XGLM-1.7B and XGLM-7.5B that they reproduced 2204.07580 tab. 7. On XWINO, mGPT-1.3B scores 56.2 zero-shot, against 54.2 for XGLM-1.7B 2204.07580 tab. 7. Few-shot NER and POS scores are far below fine-tuned encoders: 15.0 NER F1 and 24.4 POS accuracy for mGPT-1.3B at four shots 2204.07580 tab. 9 2204.07580 tab. 10. The authors say these baselines are not like-for-like, because the encoders are fine-tuned 2204.07580 §4.2.2.

Masked-entity preference score (CAMeL)

Take a prompt in Arabic in which one culture-specific entity is replaced by a mask. Compare the probability of Arab entities with the probability of Western entities at that mask. The Cultural Bias Score is the share of (Arab, Western) pairs where the Western entity wins, averaged over prompts. Zero is the ideal 2305.14456 §4.4 2305.14456 tab. 10. Card: Having Beer after Prayer? Measuring Cultural Bias in Large Language Models.

The score needs a per-token probability at the mask. Masked-language models score the full prompt, while causal and T5-type models score only the text before the mask 2305.14456 §4.4. The paper samples 50 Arab and 50 Western entities per entity type in each of five runs 2305.14456 tab. 10. Averaged over the Co prompts, scores run from 45.5 (AraGPT2 large) to 63.5 (BLOOM) 2305.14456 tab. 10. Prepending Arab demonstrations lowers the score for most models, while a culture token has little effect 2305.14456 §4.4.

Survey-persona alignment with hard and soft scores (AlKhamissi et al.)

Ask a model to answer as one real survey respondent, described by six demographic fields: region, sex, age, social class, education and marital status. Compare its answer with the answer the respondent gave in the World Values Survey, wave 7 2402.13231 §4.1 2402.13231 §4.2 2402.13231 tab. 1. Each question has four paraphrases. The model is sampled five times at temperature 0.7, and the majority answer is kept 2402.13231 §4.5. Hard alignment is exact agreement. Soft alignment gives partial credit on ordinal items 2402.13231 §4.5 2402.13231 eq. 1. Card: Investigating Cultural Alignment of Large Language Models.

Scores are averaged over the 30 questions and 303 matched personas per country 2402.13231 §4.2 2402.13231 §4.5. Averaged over the four models, soft alignment is 47.2 on the Egypt survey and 59.1 on the US survey 2402.13231 tab. 3. Hard alignment is 27.0 and 33.8 on the same surveys 2402.13231 tab. 3.

Anthropological prompting on top of persona prompts

Keep the persona and question, and add a framework of seven points from anthropological field methods. The model reasons in one paragraph under that framework, then gives the option number 2402.13231 §4.6 2402.13231 app. I. For GPT-3.5, English prompt and Egypt survey, soft alignment rises from 0.483 to 0.510 and hard alignment from 0.244 to 0.284 against vanilla persona prompting 2402.13231 tab. 4. The paper runs this on one model, one language and one survey, and does not report a variance for it 2402.13231 tab. 4.

Monolingual encoder scored on a per-language task suite (BETO and GLUES)

Pre-train a BERT-Base-sized encoder on text in one language only, then fine-tune and test it on a suite of tasks in that language. The suite, GLUES, covers natural language inference, paraphrase, NER, POS tagging, document classification and extractive QA 2308.02976 §4. The masking is whole-word, and word-level tasks use a sliding window 2308.02976 §3 2308.02976 §5.1. Card: Spanish Pre-trained BERT Model and Evaluation Data.

The baseline is the best multilingual BERT result the authors found in the literature for the same task, not a run under their setup [arxiv:2308.02976 tab. 1] 2308.02976 tab. 2. Cased BETO scores 82.01 on XNLI and 98.97 on POS. Uncased BETO scores 96.12 on MLDoc, against 95.70 for the baseline 2308.02976 tab. 1. On TAR-to-XQuAD, the baseline leads on exact match, 61.80 against 57.06 for cased BETO 2308.02976 tab. 2. The authors report these as their own numbers.

Unit-test scoring across programming languages (HumanEval-X)

Write each benchmark problem by hand in every target language, with a declaration, docstring and unit tests. The model writes the body from the declaration and docstring. The answer is scored by running the tests, and the metric is pass@k over n = 200 samples per problem, using the unbiased estimator 2303.17568 §3.1 2303.17568 §3.2 2303.17568 eq. 5. The translation task gives the model the source-language solution without its docstring, across 20 ordered language pairs 2303.17568 §3.2. Card: CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X.

The authors report average pass@1 of 18.40% on five languages, against 16.73% for CodeGen-Multi-16B 2303.17568 tab. 5. The benchmark has 164 problems in five languages, so 820 pairs 2303.17568 §3.1.

Survey and curriculum scoring for one country (KorNAT)

Split the benchmark in two. Social-value items have no single correct answer: the score for a chosen option is the share of a Korean survey that picked it. Three metrics are defined, SVA, A-SVA and N-SVA, and the last one swaps in a Neutral label for contested questions 2402.13605 §3.1 2402.13605 §4. Common-knowledge items come from Korean curriculum material and have one correct option, scored by accuracy against a reference of 0.6 2402.13605 §3.2 2402.13605 §4. Card: KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge.

The survey weights each answer for age, sex, education, residence and income 2402.13605 §3.1.4. On common knowledge, HyperCLOVA X (0.707), PaLM-2 (0.664) and Gemini Pro (0.639) are above the 0.6 reference, and the other four models are below it 2402.13605 tab. 3. On social values, PaLM-2 has the highest SVA at 0.331, against a maximum of 0.450 2402.13605 tab. 1.

Human-rated word-similarity pairs (Indian-language sets)

Translate an English word-similarity list into the target language. Two native speakers translate each pair, and a third breaks ties where they disagree 2609.34138 §4.1. Then eight native speakers rate each pair from 0 to 10, with guidelines that separate similarity from relatedness, and the score is the mean of the eight ratings 2609.34138 §4.2. The source lists are RG-65 and WordSim-353, and the six languages are Urdu, Telugu, Marathi, Punjabi, Tamil and Gujarati 2609.34138 §4.1 2609.34138 §2. Card: Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems.

To score a word-vector model, take the cosine of each pair and compute Spearman's rho against the human means, times 100. The baselines are CBOW, Skip-gram and FastText, trained on one monolingual corpus per language with 300 dimensions 2609.34138 §2. Baselines exist only for Urdu, Marathi and Telugu, while Punjabi, Tamil and Gujarati have test sets alone 2609.34138 §7. The FastText row with an OOV fallback beats plain FastText in all three, at 45.47 against 34.61 for Urdu, 38.66 against 33.68 for Marathi, and 46.02 against 34.29 for Telugu 2609.34138 tab. 2 2609.34138 tab. 3 2609.34138 tab. 4. Marathi Skip-gram scores 41.22, above the OOV row 2609.34138 tab. 3.

Distilled student for cross-lingual sentence similarity (multilingual knowledge distillation)

Start from an English sentence encoder that is already good at similarity, and use it as a fixed teacher. The student is a multilingual masked encoder. For each parallel pair, the student is trained so that both the source sentence and its translation land on the teacher's vector for the source, using a squared error 2004.09813 §2 2004.09813 §3. The authors use XLM-R as the student, because its vocabulary covers many languages 2004.09813 §3. Card: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation.

The paraphrase-trained teacher gives the best numbers. Its student averages 84.6 on monolingual STS 2017 and 83.7 on cross-lingual STS 2017, against 73.5 for LaBSE and 67.0 for LASER on the cross-lingual set 2004.09813 tab. 1 2004.09813 tab. 2. On Tatoeba search over four low-resource languages, the student scores above LASER in both directions 2004.09813 tab. 4. The authors report these numbers themselves.

Arabic encoders on a per-cluster benchmark suite (ARBERT, MARBERT and ARLUE)

Pre-train a BERT-Base encoder on one kind of Arabic text, then fine-tune it on each task in a shared suite and report one score. ARBERT is pre-trained on formal Arabic and MARBERT on tweets, which include dialects 2101.01785 §3.1 2101.01785 §3.2. The suite, ARLUE, has 42 datasets. Each fine-tuning run uses Adam, batch 32 and learning rate 2e-6 for 25 epochs, with the best epoch chosen on the dev split 2101.01785 §3.4. Card: ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.

The ARLUE score is the average over eight table rows, and MARBERT-v2 gets 77.40, against 76.55 for XLM-R Large and 76.07 for ARBERT 2101.01785 §5.2. By row, MARBERT leads on dialect region and country, while ARBERT leads on NER 2101.01785 §5.2. These are author-reported single runs.

Zero-shot chat model across seven tasks (ChatGPT, Lai et al.)

Send one prompt per example to a chat model. The prompt is a short task description, then the input, and the reply is mapped to the task's labels by rules the paper does not publish 2304.05613 §3. The authors cover POS tagging, NER, relation extraction, NLI, QA, commonsense reasoning and summarization 2304.05613 §4-2304.05613 §10. Prompt format varies by task: POS and NER take a list of words, NER asks for BIO labels, and summarization adds an output-language line 2304.05613 §4 2304.05613 §5 2304.05613 §10. Each language is placed in a resource class by its share of CommonCrawl: above 1% is high, above 0.1% medium, above 0.01% low, and the rest extremely low 2304.05613 §3. Card: ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning.

The authors give both English and target-language task descriptions for most tasks 2304.05613 §3. Averages over the languages in each task show the same pattern. POS tagging is the exception: ChatGPT with English descriptions averages 84.5 against 79.3 for XLM-R 2304.05613 §4. On the other six tasks the supervised baselines lead on average. NLI is 57.0 against 87.1 for mT5-XXL, and QA F1 is 53.5 against 85.2 2304.05613 §7 2304.05613 §8. NER macro-F1 is 29.9 against 88.4 for DAMO 2304.05613 §5. Relation extraction micro-F1 is 69.4 against 85.0 for mT5-IL 2304.05613 §6, and commonsense accuracy is 47.8 against 59.0 for TRT 2304.05613 §9. The authors report English descriptions as usually better 2304.05613 §11. These are author-reported averages over each task's language set.

Human audit of language subsets in web-crawled corpora (Kreutzer et al.)

Draw a random sample of up to 100 lines for each language subset, then have a speaker, or a rater with a dictionary and web search, label each line with a fixed taxonomy. The labels separate correct text, correct but boilerplate text, text in the wrong language, and non-linguistic content. Shares are then reported per language, macro (each language equal) and micro (weighted by size) 2103.12028 §4.1 2103.12028 §4.2 2103.12028 tab. 3. Card: Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets.

The audit covers 205 subsets of five corpora. Macro correct-share ranges from 23.74% (WikiMatrix) to 87.21% (OSCAR), and 15 subsets contain no correct sentence 2103.12028 tab. 3 2103.12028 §4.2. Labels come from one rater per line, so the authors call the per-language numbers rough 2103.12028 §4.1. The paper also tests automatic filters. CLD3 reaches 40.6% average precision on CCAligned sentence pairs, and a Transformer LangID filter raises median precision from 13.8% to 43.9% while losing 77.5% of recall 2103.12028 §4.3.

Hofstede survey items in three prompt types (Cao et al.)

Take 24 questions from the Hofstede Culture Survey and change them to third person. Prefix each with a culture phrase, and ask them in three ways: in English with a nationality, in the country's own language, and in English with "In the [country] culture setting" 2303.17466 §3.1 2303.17466 tab. 1. Re-ask until the model names an option, then map the answer to a 1 to 5 score. Six dimension scores come from a weighted difference of question pairs 2303.17466 eq. 1 2303.17466 tab. 9. Each culture's six scores are then correlated with human scores using Spearman's rank correlation 2303.17466 §4. Card: Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study.

The probe uses five cultures: US, China, Germany, Japan and Spain. Prompts 1 and 3 are both English, and they agree on over 70% of questions in every culture except China 2303.17466 tab. 3 2303.17466 §4.1. Only one Spearman cell reaches p < 0.05, Spain with prompts 1 and 2 at 0.93 2303.17466 tab. 5. The authors read the results as best alignment with the US 2303.17466 §4.2. The paper gives no human reference values in its text and no repeat count 2303.17466 app. A.1 2303.17466 §4.

Translate-then-classify for depression severity (Skianis et al.)

Translate each English Reddit post into Greek with GPT-3.5-turbo at temperature 0. Then classify both the English original and the Greek translation with one zero-shot prompt that asks for one of four severity levels, coded 0 to 3 2410.12985 §3 2410.12985 tab. 1. A reply with no recognised label counts as class 0 2410.12985 §5. The English run is the only baseline 2410.12985 tab. 2. Card: Leveraging LLMs for Translating and Classifying Mental Health Data.

On DepSeverity's 3,553 posts, the macro F1 is 0.17 in English and 0.16 in Greek 2410.12985 tab. 2. The Minimum class holds about 73% of posts, and the Minimum recall is 0.14 in English 2410.12985 tab. 1 2410.12985 tab. 2. The paper does not measure translation quality 2410.12985 §4.

When to use which

Implementation sketch

Score every language variant with the same items, then report per-language accuracy and a summary. The common core is the same for all four variants. The variants differ in how the prompt is built and whether the model is trained first.

# Score a parallel benchmark across language variants  [arxiv:2308.16884 §3]
# Input:  variants, map language -> list of items with a shared item id
# Output: accuracy per language, and the share of languages above 50

def score_parallel(model, variants, build_prompt):
    # Check that every variant holds the same item ids  [arxiv:2308.16884 §3]
    ids = None
    for lang, items in variants.items():
        lang_ids = {item.id for item in items}
        if ids is None:
            ids = lang_ids
        if lang_ids != ids:
            raise ValueError("variant is not parallel: " + lang)
    # Score each language with the same prompt builder  [arxiv:2308.16884 §4.2]
    accuracy = {}
    for lang, items in variants.items():
        correct = 0
        for item in items:
            # Choose one letter from A to D  [arxiv:2308.16884 app. A.4.2]
            prompt = build_prompt(item, lang)
            letter = model.best_letter(prompt, "ABCD")
            if letter == item.answer:
                correct += 1
        accuracy[lang] = correct / len(items)
    # Report the share of languages above 50 and 70  [arxiv:2308.16884 tab. 2]
    share_50 = sum(a > 0.5 for a in accuracy.values()) / len(accuracy)
    return accuracy, share_50

For translate-test, build_prompt first translates the passage, question and options into English 2308.16884 §4.2. For fine-tuning, model is the trained classifier and best_letter scores the four options 2308.16884 app. A.4.1. For the Cultural Bias Score, replace the accuracy with a count of pairs where the Western entity wins at the mask 2305.14456 §4.4.

Contradictions

• Translating the input. Belebele finds that translate-test beats in-language scoring for most languages it tests, on LLaMA 2-chat 70B 2308.16884 tab. 3. CAMeL argues that translated English datasets lose the cultural identity of the test data, so translation is not an effective route for cultural evaluation 2305.14456 app. A. The two papers measure different things: accuracy on a reading task, against cultural bias or fit. The translation step is also different in each. The reconciliation is not stated in either paper.
• Native-language prompting. AlKhamissi et al. find that GPT-3.5 and AceGPT-Chat fit each survey better in its dominant language 2402.13231 §5.2. This is a within-language comparison, not a translate-test one, so it does not directly test the Belebele result. Cao et al. also say most cultures align better when probed in their own language 2303.17466 §4.2, and the native-language rows of their table are higher for China, Germany, Japan and Spain 2303.17466 tab. 5. Their Spearman cells are mostly not significant, so this agrees only in direction.

Inside the Belebele paper, the text says seven languages appear in two scripts, while §3.3 names five Indo-Aryan languages and Modern Standard Arabic, which is six 2308.16884 §1 2308.16884 §3.3. The language table is the reference 2308.16884 tab. 4.

Pitfalls

• Resource classes are not reproducible. The class cut-offs are not given, the language-mention detector is unspecified, and the dataset is not yet released. No code is linked, so the paper's tables cannot be rebuilt from its text 2004.09095 §2.3 2004.09095 app. A.2.
• Baselines are literature values (BETO). The multilingual rows in its tables are the best published results per task. Dev or test is not stated, and no variance is reported, so gaps of a few tenths of a point need care 2308.02976 tab. 1 2308.02976 §5.2.
• Corpus licences (BETO). The Spanish pre-training corpus is drawn from OPUS sources, including subtitles and news. The authors say the licences of the training datasets may not be compatible with CC BY 4.0 for commercial use, so check each source before using the weights or the corpus 2308.02976 §3 README.md.
• Test set, not training set. Do not tune or train on Belebele test items 2308.16884 §3.4.
• Training set licence. The English training set is assembled from six datasets, the majority under CC BY-NC 2308.16884 app. A.3. Do not use it for commercial fine-tuning.
• Chance level depends on the protocol. The 0.25 chance level applies to classifiers. Generation with exact-match scoring has no such floor 2308.16884 §4.
• Prompt formats differ. The five-shot template uses P, Q and Answer labels. The zero-shot prompt uses a different layout 2308.16884 app. A.4.2 2308.16884 app. A.4.3. Report which one you used.
• Lenient answer parsing. The released sample code takes the first character of an unrecognised output as the answer. The paper does not describe this fallback 2308.16884 app. A.4.3 sample_zero_shot_instructions.md. Report parse-failure rates.
• Learning rate printed as a range. The XLM-V Translate-Train-All learning rate is printed as "3-6". The paper does not state the intended value 2308.16884 tab. 6.
• Translation quality. Literal translations can make the same question harder in some languages. Annotators found quality problems in the FLORES translations, and the authors changed questions to fit the translated passages instead of correcting FLORES 2308.16884 Limitations.
• Human baseline is a sample. The 97.6 human score comes from four authors answering about 30 English items each. The 95% interval covers all 900 items 2308.16884 §5.1.
• Closed models cannot be audited. GPT-3.5 Turbo's training data is not disclosed, and the paper warns that comparisons with it may be unfair 2308.16884 Limitations 2308.16884 §4.1.
• Averages hide English. LLaMA 2 70B reaches 90.9 in English but averages 48.0 across variants 2308.16884 tab. 2. Report English next to the average.
• Abstract wording versus tables (mGPT). The summary says the 13B model is on par with XGLM. Its few-shot results are below XGLM-7.5B on most tasks, so use the table 2204.07580 tab. 7.
• Perplexity normalisation conflicts (mGPT). Equation 1 normalises by characters, but the language-modelling section normalises by tokens. Compare perplexities only within one convention 2204.07580 eq. 1 2204.07580 §3.2 2204.07580 §4.1.
• Chinese is evaluated but not in the 61 languages (mGPT). The code runs Chinese on several tasks, though the paper's language list comments it out. State the language set you report on 2204.07580 tab. 1.
• CBS ranges disagree (CAMeL). The text says 40 to 60%, a figure caption says 40 to 65%, and the table caption says 30 to 80%. The table averages run from 45.5 to 63.5, so use those 2305.14456 §4.4 2305.14456 tab. 10.
• Ties and resampling (CAMeL). Ties count as zero, and each run draws new entities, so a score varies with the sample. Report the standard deviation over runs 2305.14456 §4.4 2305.14456 tab. 10.
• Hosted models change (CAMeL). GPT-3.5 is text-davinci-003 and GPT-4 is gpt-4-1106-preview. Results on these endpoints cannot be reproduced once they are retired 2305.14456 app. D.
• One vote or four paraphrases (survey). The paper majority-votes each paraphrase, but its formula reads as one vote per question and persona 2402.13231 §4.5 2402.13231 eq. 1. Decide which you implement and say so.
• Invalid answers are dropped (survey). Rows with no valid parsed option are removed, not scored as wrong, so coverage can differ between models. The paper does not report invalid-answer rates utils.py.
• Units differ between tables (survey). Tables 2 and 3 are percentages, and table 4 is fractions. The vanilla row in table 4 does not match the Egypt English row in table 2 2402.13231 tab. 2 2402.13231 tab. 4.
• "Significantly" without a test (survey). The paper calls the US gap significant but reports no statistical test. Treat the gaps as descriptive 2402.13231 §5.1.
• Licences of code and data. The mGPT repository's code licence is unclear. The AlKhamissi repository has no LICENSE file. The CAMeL prompts are derived from social-media posts and the repository is MIT. Describe these methods in your own words and check each data licence before reuse 2204.07580 §3.1 2402.13231 §4.1 2305.14456 Ethics Statement.
• Code weights are non-commercial (CodeGeeX). The model licence allows non-commercial research only and is governed by Chinese law. The repository's Apache-2.0 code licence does not extend to the weights, and the paper's CC BY licence covers the paper, not the model MODEL_LICENSE §2-§3, §6. Do not use these weights in a product.
• pass@k is reported only with enough samples (CodeGeeX). The released code reports pass@k only when every problem has at least k samples evaluate_humaneval_x.py. Fewer samples silently change which k values appear.
• Temperature for pass@10 conflicts (CodeGeeX). Section 4.1 uses 0.8, and the table caption gives 0.6 2303.17568 §4.1 2303.17568 app. A.4.1. Use 0.8 unless you are reproducing the caption.
• No reference score for social values (KorNAT). The introduction says two of seven models exceed "our reference score" on social values, but no threshold is stated. Only All-Neutral and the maximum rows are defined baselines 2402.13605 §1 2402.13605 tab. 1.
• Letter order is reversed in the harness (KorNAT). The paper lists the survey scale from Strongly Disagree to Strongly Agree. In the harness, A is Strongly Agree. Use the harness order when producing submissions kornat_eval.py 2402.13605 §3.1.3.
• Item counts differ (KorNAT). The paper reports 6,000 common-knowledge items, while the harness expects 10,000 2402.13605 app. B.2 kornat_eval.py.
• Some cross-national rows look duplicated (KorNAT). The USA prompt rows for HyperCLOVA X and Gemini Pro match in almost every cell, and the paper does not explain it. The text also says both prompts help every model except Claude-1, but Gemini Pro falls below its baseline on A-SVA and N-SVA in the table 2402.13605 tab. 2 2402.13605 §5.2.2.
• Refusals are dropped, denominators are not stated (KorNAT). The paper omits refused and invalid responses, but does not say whether they count as wrong 2402.13605 §5.4. The harness scores a social refusal as 0 README.md.
• Pair selection is not stated (word similarity). The paper names RG-65 and WordSim-353 as the bases but does not say how 97 to 163 pairs were chosen from them 2609.34138 §4.1. The sampling cannot be reproduced from the text.
• Zero vectors (word similarity). Missing words get a zero vector, and cosine with a zero vector is undefined. The paper does not say what score that gives, and it is unclear whether the rule applies to the "FastText w/ OOV" row, which generates vectors 2609.34138 §2 2609.34138 §6.
• Agreement scale (word similarity). Ratings are on a 0 to 10 scale and averaged, but agreement is reported as Fleiss' kappa. The paper does not say how the scale became categories for kappa 2609.34138 §4.2 2609.34138 tab. 1.
• Multiple choice only (KorNAT). Free-form generation was not evaluated, so the scores do not show open-ended behaviour 2402.13605 §7.
• Script teacher differs from the paper (distillation). The paper's teacher is an NLI and STS model, but the example script uses a paraphrase model and five epochs, not 20 2004.09813 §3 make_multilingual.py make_multilingual.py. Choose one setup and report it.
• Loss scale differs from the paper (distillation). The script's loss is smaller than the paper's sum by a factor tied to the embedding dimension, so a fixed loss threshold will not transfer 2004.09813 §2 embed_distill.py.
• BUCC labels are incomplete (distillation). The authors inspected 60 false positives and found 57 were valid translations that the dataset did not label as parallel 2004.09813 §4.2. Read BUCC scores with care.
• Tatoeba may flatter the student (distillation). LASER was trained on Tatoeba, and the distilled student was not, so the comparison may favour the student 2004.09813 app. A.
• The ARLUE score is not a six-cluster average (Arabic). The abstract speaks of six clusters, but the score is the average over eight table rows, because dialect is split three ways. Averaging the eight rows, each over its two metrics, reproduces the reported 77.40 2101.01785 §5.2.
• MARBERT-v2 is never defined (Arabic). The results table reports it, but the paper does not say how it differs from MARBERT. Treat the name as a label only 2101.01785 §5.2.
• Arabic checkpoints are for research (Arabic). Do not ship them in a commercial product on the strength of this concept page README.md.
• Model and settings not pinned (ChatGPT). The paper names "ChatGPT" and the GPT-3.5 series, with replies collected from 1 March to 5 April 2023. Sampling settings, max tokens and the snapshot are not stated, and the interface (web or API) is not named 2304.05613 §1 2304.05613 §3. Hosted models change, so these numbers cannot be rerun exactly.
• Parse rules are not published (ChatGPT). The paper's formula nests the reply step inside the parser, and the parse rules are described only as task-specific. Parse failures cannot be separated from model errors 2304.05613 §3.
• Language count differs (ChatGPT). The method says 34 languages and the introduction and conclusion say 37 2304.05613 §3 2304.05613 §1. Use the per-task tables, not the headline count.
• Corpus audit labels come from one rater (web corpora). Each sentence has one label, and some raters had no speaker of the language. The authors put accuracy against expert labels at 0.66 on six classes for CCAligned, 0.79 on two classes 2103.12028 tab. 4 2103.12028 §4.1. Dialects were not handled 2103.12028 §7.
• Macro and micro give different answers (web corpora). CCAligned is 29% correct macro but 54% micro, and OSCAR is 87% macro and 99% micro 2103.12028 tab. 3. Say which one you quote.
• Language codes must be checked (web corpora). The paper finds 83 nonstandard codes, and at least 48 of the 83 affected corpora are entirely spurious. It names deprecated codes such as sh and iw 2103.12028 §5 2103.12028 app. A.
• Repeat count and sample size are missing (Hofstede probe). The number of re-asks, the decoding settings and the model version are not stated. The Spearman sample is not stated either. Six dimensions per culture would give n = 6, which is an inference that the p-values depend on 2303.17466 §4 2303.17466 tab. 5.
• Table 4 rows for uai and mas look swapped (Hofstede probe). Applying eq. 1 with the paper's coefficients to the Prompt 1 scores gives, for uai, the values printed in the mas row, and the reverse for mas. Check the rows before you rely on them 2303.17466 tab. 4 2303.17466 tab. 9 2303.17466 tab. 10.
• Scale direction varies (Hofstede probe). Some questions list "always" first and others "never" first, and the paper does not say whether any scores were reversed 2303.17466 app. A.1. Use the printed option numbers.
• Free text is mapped to options without a rule (Hofstede probe). Half-point scores come from averaging several named options, and no inter-rater check is reported 2303.17466 tab. 10.
• No scoring code for the Hofstede probe. The repository the paper links holds data files and a README, with no scoring script and no licence, so the scores cannot be checked against code 2303.17466 app. A.5. Re-derive them from the paper's tables.
• Greek prompt is not stated (translate-then-classify). The paper does not say whether the classification prompt was also translated for the Greek run. Choose one and report it 2410.12985 §4 2410.12985 tab. 1.
• Translation quality is not measured (translate-then-classify). The only check is a qualitative look at two examples 2410.12985 §4. The Greek scores mix classifier and translation effects.
• Minimum class dominates (translate-then-classify). The Minimum class has 2,587 of 3,553 posts. Its recall is 0.14 in English, so the model rarely predicts it, and the macro average weights the rare classes equally 2410.12985 tab. 1 2410.12985 tab. 2. Report per-class scores with the macro.
• Macro row is rounded differently (translate-then-classify). The mean of the four English F1 values is about 0.178, and the table prints 0.17. The paper does not say how the macro row was computed 2410.12985 tab. 2.
• Low-resource replies often fail (ChatGPT). For Thai, Bengali and Burmese, only 42 to 59 percent of summarization requests returned a non-empty reply with English descriptions, so some low scores reflect failed requests 2304.05613 §10.

Papers

• Joshi et al., language resource classes in NLP venues: six language resource classes, and how inclusive NLP venues are toward each class.
• Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation: an English sentence encoder distilled into a multilingual student, scored on cross-lingual STS, BUCC and Tatoeba, with author-reported comparisons to LASER, mUSE and LaBSE.
• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic: two Arabic BERT-Base encoders and the 42-dataset ARLUE suite, with author-reported scores.
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants: a 900-question parallel reading set in 122 variants, with four protocols and author-reported results for each.
• mGPT: Few-Shot Learners Go Multilingual: two decoder-only models in 61 languages, scored by label likelihood across classification, NER, POS and probing.
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models: the CAMeL Arabic prompt set and the Cultural Bias Score for Arab versus Western entity preference.
• Investigating Cultural Alignment of Large Language Models: persona-based simulation of the World Values Survey in Egypt and the US, with hard and soft scores.
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning: zero-shot ChatGPT on seven NLP tasks in 34 to 37 languages, with author-reported comparisons to supervised baselines.
• Spanish Pre-trained BERT Model and Evaluation Data: a Spanish-only BERT-Base encoder and GLUES, a Spanish task collection, with author-reported comparisons to multilingual BERT.
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X: a 13B code model and HumanEval-X, a five-language unit-test benchmark, with author-reported pass@k.
• Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems: six new word-similarity sets built by translation and native-speaker rating, with CBOW, Skip-gram and FastText baselines for three languages.
• KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge: a Korean benchmark that scores social-value answers against a national survey and common knowledge against school curricula.
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets: a human audit of 205 language subsets from five web-crawled corpora, with an error taxonomy and tests of automatic LangID filters.
• Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study: Hofstede survey items asked in three prompt types for five cultures, with author-reported Spearman correlations to human scores.
• Leveraging LLMs for Translating and Classifying Mental Health Data: GPT-3.5-turbo translates Reddit posts into Greek and classifies depression severity, with author-reported macro F1 in English and Greek.