Large Language Models Are State-of-the-Art Evaluators of Translation Quality

In plain words

Translation software is usually graded in one of two ways: by comparing its output with a human translation, or with a learned scoring model trained on human ratings. This paper asks whether a general chatbot-style language model can do the grading instead. It asks a GPT model to rate one translated sentence at a time, with a plain instruction and no worked examples. The rating can be a number from 0 to 100, a rating of one to five stars, or one of five quality labels. The model may also be shown the human reference translation, or not.

A whole test set gets one score: the average of its sentence scores. On a 2022 benchmark covering three language pairs, the best version with the reference beat every metric in the authors' main comparison table. Smaller GPT models were close to random or did not give usable answers, so the approach only works with the larger ones. The authors stress that the results are promising but limited to three language pairs and to whole-test-set scores.

TL;DR

• GEMBA scores each segment alone with a zero-shot prompt to a GPT model, then averages the segment scores per system §2 §2.2.
• Four prompt types (direct assessment, scalar quality, stars, classes), each with or without a human reference §2.1.
• On WMT22 MQM (en-de, en-ru, zh-en), GEMBA-GPT4-DA with reference reaches 89.8% system-level pairwise accuracy, against 85.0% for the non-public MetricX XXL tab. 2.
• Only GPT-3.5-class and larger models work; Babbage and Curie are "close to random", and GPT-2 and Ada give no usable scores §4.3.
• At segment level the same GPT-4 prompt reaches Kendall tau-b 0.36 to 0.38, against 0.36 to 0.43 for the best metrics in the table tab. 4.
• All numbers are author-reported.

Problem

Automatic translation metrics are needed for system comparison and development. The authors' starting point is that a model able to translate may also be able to tell good translations from bad ones §1. The question is whether pre-trained generative models can assess translation quality with zero-shot prompts alone §1.

Assumptions and risks the paper names:

• Only GPT-3.5 and larger models produced usable scores [abstract] §4.3.
• Evaluation covers only three language pairs, and under-resourced languages may score worse [Limitations].
• Segment-level performance is weaker than the best metrics [Limitations] §4.4.
• OpenAI's training data may overlap the test set. The authors call this unlikely but cannot rule it out, and the test set is not available in plain text [Limitations].
• Human MQM labels are the gold standard, and they are noisy, so 100% accuracy is not reachable §3.1 §4.1.

Method

GEMBA is a prompt-based metric. For each segment, the scorer fills one fixed zero-shot template with five fields: the prompt variant, the source language name, the target language name, the source segment, and the candidate translation. The reference translation is added only in reference mode §2. Each segment is one independent request. Few-shot prompting and document context are left to future work §2 §5.

Prompt variants §2.1 app. A:

1. Direct assessment (DA): a score from 0 to 100, where 0 means "no meaning preserved" and 100 means "perfect meaning and grammar". This is the prompt the authors call best-performing and show as fig. 1.
2. Scalar quality metric (SQM): a score from 0 to 100 with anchor phrases on the way from "No meaning preserved" to "Perfect meaning and grammar" app. A.2.
3. Stars: one to five stars with a description for each star app. A.3.
4. Classes: one of five labels, mapped to the integers 0 to 4 in label order before averaging §2.2 app. A.4.

Each variant has a reference mode (plain name) and a quality-estimation mode without a reference (suffix [noref]) §2.1.

Models §2.3 tab. 1: GPT-4 is the default. The paper lists nine models from GPT-2 to GPT-4. It reports seven; GPT-2 and Ada are left out because they gave no usable scores §4.3.

Scoring §2.2 §4.5: if an answer is not a valid value for its prompt, the model is queried again at a higher temperature until a valid answer appears. The paper does not give the exact schedule. Invalid answers are rare for the main models tab. 6.

# Score one segment with the zero-shot DA prompt  [§2] [§2.1]
# Input:  source, hypothesis, reference (None in quality-estimation mode)
# Output: integer 0-100, or None if no valid answer was found

def score_segment(source, hypothesis, reference, source_lang, target_lang):
    # 1. Fill the zero-shot template; the reference line is ref mode only
    #    [app. A.1]
    prompt = fill_da_template(
        source, hypothesis, reference, source_lang, target_lang
    )
    # 2. Attempt k sends temperature k / 10, so 0.0 up to 1.0  [§4.5]
    #    [code: gemba/gpt_api.py#L78-L86]
    for step in range(11):
        temperature = step / 10
        # 3. One completion, no top-p or penalty changes  [§4.5]
        #    [code: gemba/gpt_api.py#L136-L143]
        reply = call_model(prompt, temperature)
        # 4. A cut-off reply is invalid when no max_tokens is set
        #    [code: gemba/gpt_api.py#L116-L122]
        if reply.finish_reason == "stop":
            # 5. Accept one integer in 0..100 only  [§2.2]
            #    [code: gemba/prompt.py#L15-L39]
            score = parse_integer_in_range(reply.text, 0, 100)
            if score is not None:
                return score
    # 6. No valid answer after all attempts: the segment stays unscored
    #    [code: gemba/scores.py#L41-L55]
    return None

# System-level score = mean of valid segment scores  [§2.2]
# Input:  one score per segment (None when unscored)
# Output: one float for the system

def system_score(segment_scores):
    # 1. Drop unscored segments; they are not counted as zero
    #    [code: gemba/scores.py#L85-L92]
    valid_scores = [s for s in segment_scores if s is not None]
    # 2. Average what remains. Classes are already indices 0-4  [§2.2]
    return sum(valid_scores) / len(valid_scores)

Accuracy §3.2. The paper defines system-level pairwise accuracy as the fraction of system pairs whose metric difference has the same sign as the human difference. Segment-level agreement uses Kendall's tau; the text says the tau-b variant is used, and it gives a formula without tie adjustment §3.2.

# System-level pairwise accuracy  [§3.2]
# Input:  one metric score and one human score per system
# Output: fraction of system pairs ordered the same way by both

def pairwise_accuracy(metric_scores, human_scores):
    # 1. Visit each pair of systems; the paper does not say whether
    #    ordered or unordered pairs are counted  [§3.2]
    agreeing = 0
    pair_count = 0
    for i, j in system_pairs(len(metric_scores)):
        # 2. A pair agrees when both differences have the same sign
        metric_sign = sign(metric_scores[i] - metric_scores[j])
        human_sign = sign(human_scores[i] - human_scores[j])
        if metric_sign == human_sign:
            agreeing += 1
        pair_count += 1
    return agreeing / pair_count

Hyperparameters

Results

Author-reported. Setup: WMT22 MQM test set for three language pairs; GPT models queried zero-shot through the OpenAI or Azure API §3.1 gpt_api.py; hardware and cost not stated. Baselines are the WMT22 metrics §3 tab. 2.

Complete tables from the paper follow.

tab. 1 Models used. The API names are as printed in the paper.

Caption: "GPT-2 and Ada do not work."

tab. 2 System-level pairwise accuracy against the WMT22 metrics. Metrics marked [noref] use no reference.

tab. 3 System-level accuracy by prompt and model (quality-estimation prompts for most combinations). The caption says results above MetricX XXL are bolded in the original.

tab. 4 Segment-level Kendall tau-b for the main metrics, per language pair.

tab. 5 Share of all distinct segment scores, GPT-4, by value (per the caption, for model GPT-4). Columns: DA, DA[noref], SQM, SQM[noref].

tab. 6 Number of invalid answers that were re-prompted with added randomness (full set 106,758 segments). Caption: ChatGPT and parts of GPT-4 were excluded. --- is not explained in the paper.

tab. 7 Complete results: system-level accuracy and segment-level Kendall tau-b for every metric and prompt (appendix B).

Paper vs code

The official repository is linked from the abstract (footnote 1) and belongs to the Microsoft organisation the authors work for [abstract] [Acknowledgments]. Its code, prompts and scores are under CC BY-SA 4.0 LICENSE.md README.md. That is a share-alike licence, so no code is quoted verbatim here; the pseudocode above is written from the cited lines.

Differences between paper and code:

• Model in the shipped experiment script. The paper's default is GPT-4 §2.3. The script that collects the WMT22 DA scores lists only text-davinci-003 gemba_da.py. The README recommends gpt-4 for GEMBA-DA README.md. Reproducing the GPT-4 numbers means changing the model name and the run loop.
• Script does not start as shipped. gemba/gemba_da.py checks credentials["deployments"] gemba_da.py, but credentials is never defined in the file or the package. The first run therefore raises an error at that line gemba_da.py.
• Prompt wording differs from app. A. The reference-mode DA template drops "the" before "human reference" and labels the candidate "machine translation" prompt.py app. A.1. The classes template labels the candidate "machine translation" prompt.py while app. A.4 says "translation". Use the code's wording if you want to match the released numbers, and say so.
• Retry behaviour. §2.2 says the authors "sample more responses". The code asks for exactly one completion per attempt and changes the temperature in 0.1 steps gpt_api.py gpt_api.py.
• Unscored segments. The paper notes some segments have no score even after retries app. B caption. The code stores them as None and leaves them out of system means scores.py.
• Class values. The paper maps classes to 0-4 §2.2. The code returns the index of the matched label in the list prompt.py.
• Evaluation. The evaluation script calls the mt-metrics-eval eval_metrics with gold mqm, system level, on the three pairs evaluate.py. This matches the paper's "official WMT22 script" §3.2.
• Endpoint. The code uses Azure OpenAI when OPENAI_AZURE_ENDPOINT is set, else the OpenAI API with OPENAI_API_KEY gpt_api.py.

Pitfalls

• Model count. §2.3 says "seven GPT models", but its list, tab. 1 and the abstract cover nine. Use tab. 1.
• Parsing numbers. §4.5 says the authors "parse only the first number". The code takes an answer only if it contains exactly one run of digits, or one of two fixed forms (a bracketed number, or N/100) prompt.py. "85.5", "90 (previously 80)" and "90 out of 100" are all invalid and trigger a retry.
• Star answers in other languages. §4.5 says answers may come in the target language, with examples "一颗星" and "五". The star parser knows English number words, digits and the star symbols only, so such answers are rejected and re-sampled prompt.py. The paper's en-de and en-ru runs are affected if the model answers in German or Russian.
• Contradictory stars. If a reply names two different star counts, the star parser returns nothing prompt.py. This is safe but costs retries.
• Two class labels. A reply that contains two class labels is invalid prompt.py. Labels are matched as lowercase substrings.
• Truncated replies. Without max_tokens, a reply that ends for any reason other than "stop" counts as no answer gpt_api.py. The shared scorer sets 500 and extends by 200 per cut-off utils.py gpt_api.py. The 100-token limit declared for classes in prompt.py is not passed on by either path.
• Failure-rate text. §4.5 says invalid answers are below 1% except for SQM, but tab. 6 shows Curie DA with 8,048 invalid answers (about 7.5% of 106,758) and Babbage Stars with 18,074 (about 17%). The main models, GPT-4 and Davinci-003, stay small. GPT-4 counts are partial tab. 6 caption.
• Choice of headline prompt. Table 3 and tab. 7 show GPT-4 Stars and GPT-4 Classes[noref] at 91.2%, above the DA headline of 89.8%. The text names DA as "the main style" for some experiments and does not explain the choice §4.
• Score-distribution text. The text gives 60.5% for "95" under DA[noref]; tab. 5 gives 60.6%. The text also says over three-quarters of DA scores are 80, 95 or 100; tab. 5 sums to 68.2% for those values.
• Segment-level formula. §3.2 prints the simple (concordant minus discordant) over (concordant plus discordant) formula, but the text says tau-b, which adjusts for ties. The tie-adjusted formula is not printed. Ties matter because the score is discrete; §4.4 conjectures that ties lower the segment scores, without testing it.
• System-level ties. The paper states there are no system-level ties in its data §4.4. It does not say how a zero difference is signed in accuracy.
• Determinism. Temperature 0 was re-run for one configuration only §4.5. The code caches responses by model, temperature and prompt gpt_api.py, so a re-run can read stored answers rather than query the API again.
• Model snapshots. tab. 1 names 2023 API model ids. The paper does not say whether they can still be queried or behave the same way. Check before reuse.
• Contamination and scope. The authors cannot exclude that the WMT22 test data was in OpenAI training data, and they say the test set is not in plain text [Limitations]. Results cover three language pairs and system level only [Limitations].

Lineage

• Builds on: the DA and SQM human-assessment templates of the Appraise framework used for WMT22 §1. Cited by name in the paper, with no arXiv id given.
• Related GPT translation study: 2302.09210, which the authors cite for Davinci-003 translation strength §4.3.
• Concurrent work: Lu et al. (EAPrompt), named in the paper §1, with no arXiv id given.
• Compared against: WMT22 Metrics shared-task metrics, including COMET, BLEURT, MetricX XXL and UniTE §3 tab. 2.
• Future directions named by the authors: few-shot prompting, fine-tuning, MQM error-based and post-editing prompts, and document-level evaluation §5.