ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

In plain words

The paper asks whether a chat assistant that works well in English also works well in other languages. The authors test ChatGPT on seven common language tasks, such as tagging the parts of speech in a sentence, finding names of people and places, answering questions from a passage, and judging whether one sentence follows from another. The languages range from widely spoken ones to rare ones with little text on the web.

They give the model plain instructions with no worked examples, the way an ordinary user would, and score its answers against standard test sets. The answer is mostly negative: ChatGPT usually falls well short of models that were trained for each task. It also tends to do better when the instruction is in English, even when the text is in another language. Part-of-speech tagging is the clear exception. The results describe one model as it behaved between March and April 2023, and the authors plan to keep extending the study.

TL;DR

• Zero-shot ChatGPT (one prompt per example, no examples shown) is evaluated on seven tasks in 34 to 37 languages. The paper states both counts §3 §1.
• Against supervised baselines, ChatGPT is far behind on most tasks. POS tagging is the exception, where it is competitive or better §4 §11.
• English task descriptions usually beat target-language descriptions §11.
• Every number is author-reported, and the model snapshot is not named §3.

Problem

Most ChatGPT evaluations up to this point used English data. Earlier multilingual checks covered only a few tasks, few languages, or small samples §1. The authors argue that it is unclear whether a multilingual LLM can be relied on for many languages, or whether language-specific systems are needed §1.

The paper also targets reproducibility. Because ChatGPT is large and not fine-tunable by most users, it is evaluated only in the zero-shot setting, with single-stage prompts that mirror ordinary use §3.

Method

ChatGPT is treated as a black box. For each example the paper builds one prompt and parses the free-text reply into the label format of the task §3.

The steps the paper describes:

1. Choose tasks and datasets. Seven tasks: POS tagging, NER, relation extraction (called "relation classification" in one place), NLI, QA, commonsense reasoning, and summarization §3 §4-§10. Each task uses an existing multilingual dataset (section details below).
2. Assign resource levels by web share. A language's category is set by its share of the CommonCrawl corpus, the main pre-training data of GPT-3 §3. The thresholds are in the pseudocode below. The full language table is in Results.
3. Write a task description D. It is kept simple and states the label set and the expected output format §3 §4. Descriptions are given in English or in the target language. Non-English descriptions were produced with an automatic translation tool §3.
4. Build the prompt. The prompt is the description followed by the input X. Some tasks add an output-format note or extra fields (see steps 5 to 10) §3.
5. POS tagging. The sentence is given as a list of words, not raw text, so that the labels can be aligned to words. The reply must be a list of (word, tag) tuples §4.
6. NER. Same structure as POS. Labels are requested in BIO form, one per word, with six entity types. The authors report that BIO requirement gives better scores than asking for names without BIO §5.
7. Relation extraction. The prompt has the description (listing all relation types), the text, and the two entity mentions. The reply must read "Relation: " §6.
8. NLI. Description, premise, hypothesis, then a multiple-choice question over entailment, contradiction and neutral §7.
9. QA. The English description is taken from NaturalInstructions. A note asks for a short answer copied from the passage. The reply's trailing period is removed before scoring §8.
10. Commonsense reasoning. Description, question and lettered options. For X-CSQA the description names the input language. For the Wikipedia cloze set, a masked sentence and four options are given §9.
11. Summarization. The prompt is "summarize this text" plus an output-language instruction, so that the summary is in the input language §10.
12. Parse and score. Replies are parsed with task-specific rules that the paper calls P, and scored with the task metric §3. The paper does not publish these rules.

Pseudocode for a single query, as the paper describes it:

# Zero-shot query with one prompt per example  [§3]
# Input:  task description D, example input X, parse rules P, model G
# Output: prediction Y

def predict(task_description, example_input):
    # Concatenate the description and the input  [§3]
    prompt = task_description + example_input
    # Start a fresh conversation for each query  [§3]
    reply = G(prompt)                       # R = G([D;X]); model not named
    # Map the free-text reply to the label format of the task  [§3]
    prediction = P(reply)                   # rules not published
    return prediction

Resource level, as the paper uses it:

# Resource level from the share of CommonCrawl  [§3] [tab. language in §3]
# Input:  cc_share, the percentage of the corpus for the language
# Output: "high", "medium", "low" or "extremely low"

def resource_level(cc_share):
    # Above 1 percent is high  [§3]
    if cc_share > 1:
        return "high"
    # Above 0.1 percent and up to 1 percent is medium  [§3]
    elif cc_share > 0.1:
        return "medium"
    # Above 0.01 percent and up to 0.1 percent is low  [§3]
    elif cc_share > 0.01:
        return "low"
    # Below 0.01 percent is extremely low  [§3]
    return "extremely low"

Hyperparameters

The paper reports no model hyperparameters. The generation settings are not stated, because the chat interface was used and its sampling settings are not given.

Results

All numbers are author-reported. Supervised baselines are the models the paper names as state of the art for each task. Whether they were re-run or copied from earlier papers is not stated for every task. For NLI the paper says the mT5 numbers are the best reported values §7.

Tasks and data (setup).

• POS: XGLUE-POS (Universal Dependencies tags), 17 languages (Portuguese dropped because the Hugging Face copy lacks it), test sets with more than 15K samples for the selected languages §4.
• NER: MultiCoNER, 11 languages, test sets, six entity types §5.
• RE: SMiLER, 14 languages, 36 relation types including no-relation, test sets with more than 12K samples §6.
• NLI: XNLI development set, 15 languages including English §7.
• QA: XQuAD, 240 paragraphs and 1,190 question-answer pairs in English and ten translations §8.
• CSR: X-CSQA development set (1,000 per language) and Wikipedia cloze QA from IndicNLPSuite (62,314 samples in total) §9.
• Summarization: XL-Sum, 12 selected languages, test sets §10.

Language resource categories (from the CommonCrawl share; population columns omitted from this copy) §3:

POS tagging, accuracy §4. Supervised baseline is XLM-R. ChatGPT is given in English (en) and target-language (spc) descriptions.

NER, macro-F1 §5. Supervised baseline is DAMO.

NER per label, English prompts, with spurious share of predictions §5:

Relation extraction, micro-F1 §6. Supervised baseline is mT5-IL (base).

NLI, accuracy on XNLI dev §7. Supervised baseline is mT5-XXL.

QA on XQuAD, EM / F1 §8. Supervised baseline is mT5-XXL.

Commonsense reasoning, accuracy §9. Left: X-CSQA dev, supervised baseline TRT, ChatGPT in English (en) and target-language (tgt) descriptions.

Right: Wikipedia cloze QA (IndicNLPSuite), supervised baseline IndicBERT §9.

Summarization, ROUGE §10. ChatGPT with English descriptions. Lengths are averages in characters. Success is the share of requests that returned a non-empty reply.

Same setup with language-specific descriptions §10:

Headline claims, as the paper states them §11 §12: ChatGPT is significantly worse than state-of-the-art supervised models on most tasks and languages, POS tagging is the exception, and English descriptions are generally better. The paper also reports that for POS tagging, ChatGPT beats XLM-R on 13 of 17 languages §4. By our check this count holds only if the better of the two prompts is taken per language. With English descriptions alone it is 12, and with target-language descriptions alone it is 9.

Paper vs code

No official code.

Searched: (1) the paper text, abstract and LaTeX source, for repository or project links, found none (the only GitHub links are to a third-party scorer and a third-party scoring tool); (2) the GitHub repository search for "chatgpt beyond english", "chatgpt multilingual zero-shot" and "ChatGPT multilingual evaluation XGLUE SMiLER", and for the author surname with ChatGPT, found no repository from the authors. No discrepancies can be checked against code.

Pitfalls

• Language count disagrees within the paper. The methodology says 34 languages §3; the introduction, discussion and conclusion say 37 §1 §11 §12. By our count of the language codes that appear in the results, 36 are used, and Indonesian (id) is in the language table but in no result table §3 §4-§10. Plan on the per-task tables, not the headline count.
• Task name differs. The methodology calls the task "Relation Classification" §3, while the other sections say "Relation Extraction" §1 §6 §12. The dataset is SMiLER, with a no-relation option §6.
• The model is not pinned. The paper says "ChatGPT" and "GPT-3.5 series" §1, with responses from 1 March to 5 April §3. Snapshot, sampling settings and max tokens are not stated. Hosted models change, so the numbers cannot be reproduced exactly even with the prompts.
• Interface not stated. The method states that conversations were cleared for each query §3, but it does not say whether this was the web app or the API. Implementers who call an API should expect different behaviour from a web session.
• The parse function is not published. The formula is written as Y = P(R(G([D;X]))), which nests R inside P even though R is defined as G([D;X]) §3. The evident intent is Y = P(R). The parse rules P, which turn free text into labels, are only described as "pre-defined task-specific rules" §3. Parsing errors therefore cannot be separated from model errors.
• Prompt construction varies per task. POS uses word lists §4. NER requires BIO output §5. QA adds a note and removes the trailing period §8. Summarization adds an output-language line §10. Anyone re-implementing one task should copy that task's prompt, not a single generic template.
• Units and sample sizes are unclear. POS test sets are described as "more than 15K samples" without per-language counts §4. Summarization lengths are in characters §10. NER sample counts are not given §5.
• Chinese summarization lengths conflict between tables. The Chinese gold length is 420.10 in the English-description table, but 55.31 in the target-description table, and the model length swaps accordingly §10. The claim that ChatGPT summaries are generally longer than gold summaries §10 does not hold for Chinese with English descriptions (191.46 against 420.10).
• Relation extraction text contradicts its table. The text says ChatGPT performs worst in English for relation extraction §6. The table has Persian lower than English on ChatGPT (en) (58.9 against 61.9) §6.
• Wikipedia cloze resource level conflicts. The text calls the 11 languages "low- and extremely-low-resource" §9, but the table lists Hindi as medium §9. Read the table's categories, not the text's wording, when grouping the languages.
• Resource categories are proxies. Categories depend on CommonCrawl share, which the authors say is not the only factor in performance §11. Boundary values (exactly 1%, 0.1%, 0.01%) are not stated §3. The Japanese code jp is not ISO 639-1 (ja) §3 §9.
• Translations were not validated. Non-English descriptions were made by an automatic tool §3. Whether they were checked by native speakers is not stated.
• Appendix is not in the compiled paper. The source tree has a language- specific prompt appendix, but the main file does not include it §3. Treat its contents as not part of the paper.
• Baselines differ in provenance. For NLI the paper says the mT5 numbers are the best reported values §7. For other tasks it names the baseline as state of the art without saying whether it was re-run §4-§10.
• Zero-shot only. No chain-of-thought, self-consistency or few-shot condition is tested §3 §11, so the numbers say nothing about those methods.
• Summaries are long. The authors attribute part of the summarization gap to long output §10. Length control is not a tested variable.
• Lower-resource success rates are low. For Thai, Bengali and Burmese the share of non-empty replies is 42 to 59 percent on the English-description table §10, so a low score may partly reflect failed requests.

Lineage

• Compared against: XLM-R §4, DAMO §5, mT5-IL §6, mT5-XXL §7 §8, TRT §9, IndicBERT §9 and the supervised XL-Sum baseline cited in §10.
• Mentioned, not evaluated: BLOOM, GPT-4 and BARD, which the authors name as future comparisons §11.
• Builds on: the evaluation literature on ChatGPT that the introduction surveys §2. No single prior method is built on.