How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

In plain words

Machine translation software has long been built as a dedicated system that reads a sentence in one language and writes it in another. Large chatbot-style language models can also write translations, just by being asked in plain text. This paper asks how good three OpenAI GPT models are at that job, compared with the best dedicated translation systems and a commercial translator.

The authors test many language pairs, including some with little training data. They also try showing the model a few example translations before the real request, and feeding it a whole document instead of one sentence at a time. A last experiment combines the two worlds: a normal translator produces the output, and the GPT model is used only when a quality estimate says the output is weak. The main finding is that the models translate well for widely spoken languages, struggle with rare ones, and that the usual word-overlap scores (BLEU, chrF) understate their quality, so the authors lean on neural quality metrics and human judgements.

TL;DR

• Three GPT models are compared on 18 translation directions from WMT test sets; text-davinci-003 is the main model for later experiments §2.3 §3.3.
• With 0, 1 or 5 example pairs in the prompt, text-davinci-003 is close to or above the WMT best systems on most high-resource directions, and clearly weaker on Icelandic and Hausa tab. 3 tab. 4.
• Document-level prompting raises COMET scores and cuts the number of API calls §3.5 tab. 5.
• A hybrid with Microsoft Translator as primary and GPT as fallback gives the highest COMET-22 scores, up to 1.6 points above the better single system §3.7 fig. 1.
• All numbers are author-reported. The paper states no sampling settings for the GPT API calls.

Problem

Neural machine translation (NMT) systems are trained on large parallel corpora and use an encoder-decoder design §1. GPT models are decoder-only, trained mostly on monolingual text with a strong English bias, and need many parameters to work across languages §1. Before this study, GPT translation quality against the strongest NMT and commercial systems was little studied §1.

Assumptions and risks the paper names:

• Training-data overlap: the authors state that the GPT models' training data was collected up to June 2021, so they use WMT22 test sets (and WMT21 for Icelandic and Hausa), which are newer and use natural source texts §2.1.
• Metric reliability: lexical metrics (BLEU, chrF) are said to give misleading signals for GPT output, and document-level metrics are limited §3.2 [Limitations].

Method

The paper is an evaluation study, not a new model. Each part below describes how the authors ran one experiment. Section numbers are from the compiled paper.

Systems. text-davinci-002 (an InstructGPT model), text-davinci-003 (described as an improved text-davinci-002) and ChatGPT, all called through the Azure OpenAI Service §2.3. Baselines are the WMT-best system per direction and Microsoft Translator via Azure Cognitive Services §2.2 tab. 1.

Metrics. COMET-22 (wmt22-COMET-da) as the reference-based metric, COMETkiwi (wmt22-COMETkiwi-da) as the reference-free quality estimate, plus SacreBLEU and chrF §2.4. Human evaluation uses source-based contrastive Direct Assessment plus Scalar Quality Metric (contrastive DA+SQM) §2.4 §4.

Shot selection §3.1. The shot pool is the WMT training data for each direction, cleaned by language identification and length ratio §3.1 app. D. Three strategies are compared:

• Random (RR): shots drawn from the full cleaned pool.
• Quality random (QR): shots drawn at random from the 1,000,000 pairs that LaBSE ranks highest.
• Quality selected (QS): shots chosen by similarity to the input, from the quality pool, as described below.

The paper tests 0, 1 and 5 shots and reports that more than 5 gave no meaningful gain in preliminary runs §3.1.

# Quality-selected (QS) shots  [§3.1]
# quality_pool: LaBSE-ranked training pairs, top 1,000,000 per direction
# k: number of shots, 1 or 5

def select_relevant_shots(source, quality_pool, k):
    # Stage 1: lexical retrieval with BM25 (Elasticsearch)  [§3.1]
    candidates = bm25_top(quality_pool, query=source, n=64)
    # Stage 2: rank candidates by LaBSE cosine distance to the input  [§3.1]
    distances = [
        cosine_distance(labse(source), labse(pair.source))
        for pair in candidates
    ]
    # Keep the k closest pairs as the few-shot examples  [§3.1]
    return take_k_nearest(candidates, distances, k)

Prompt format. Sentence-level prompts follow the instruction format the OpenAI playground recommends for translation: an instruction line, then pairs of source and numbered reference, then the input and an empty numbered slot app. C. ChatGPT zero-shot uses a different template, with a ### Translate this sentence from X to Y, Source: header and a ### Target: slot app. C.

Document-level translation §3.5. The whole document, or a window of sentences, is sent in one request with a prompt that asks for one output line per input line app. C. The model's output is then realigned to the source: line breaks are inserted where a source sentence was split across lines, and empty lines replace skipped sentences §3.5. Document windows of 2, 4, 8, 16 and 32 sentences are compared in zero-shot mode, and a window of 10 sentences is used for the few-shot document study §3.5.

Document metric (Doc-COMETkiwi) §2.4. COMET cannot align a document sentence-for-sentence once the output is restructured, so the authors slide an overlapping window over the document and average the segment scores. Each sentence is scored in several contexts. The paper does not give the window size used for the reported Doc-COMETkiwi values.

# Doc-COMETkiwi: overlapping-window quality estimate  [§2.4]
# source_doc, hyp_doc: documents aligned sentence by sentence  [§3.5]

def doc_comet_kiwi(source_doc, hyp_doc, window):
    # Slide a window over the document, one sentence at a time  [§2.4]
    segment_scores = []
    for center in range(len(source_doc)):
        # Context around the current sentence, limited to `window` sentences
        # [§2.4] [code: tools/doc_score.py#L19-L27]
        src_context = build_context(source_doc, center, window)
        hyp_context = build_context(hyp_doc, center, window)
        # Reference-free COMETkiwi on the joined segments  [§2.4]
        segment_scores.append(cometkiwi(src_context, hyp_context))
    # Average over all segments to score the document  [§2.4]
    return mean(segment_scores)

Hybrid routing §3.7 fig. 1. Microsoft Translator is the primary system. GPT is used as a fallback. Two variants:

# Hybrid "threshold" routing  [§3.7]
# The threshold is the median COMETkiwi score of MS-Translator output

def hybrid_threshold(source, ms_output, threshold):
    # Estimate the quality of the primary output without a reference  [§3.7]
    quality = cometkiwi(source, ms_output)
    # Re-translate with GPT only when the primary output is below threshold
    # [§3.7]
    if quality < threshold:
        return gpt_translate(source)
    return ms_output

Max-routing is an upper bound, not a deployable method. For each sentence it keeps whichever of the two outputs has the higher COMETkiwi score §3.7.

Hyperparameters

Results

Author-reported. Setup: WMT21 and WMT22 test sets for 18 directions; text-davinci-003 unless noted; baselines are WMT-best systems and Microsoft Translator §2.1 §2.2 §3.3. Hardware: not stated (API access).

Headline, COMET-22 / COMETkiwi. For each direction the GPT column shows the best few-shot setting by COMET-22 among the GPT rows the paper marks with an asterisk. tab. 3 and tab. 4.

Findings the paper states:

• Zero-shot, text-davinci-003 is the best of the three GPT models in most directions; text-davinci-002 is weakest in every pair shown §3.2 tab. 2. On DE-EN, COMET-22 is 73.2, 84.8 and 84.8 for text-davinci-002, text-davinci-003 and ChatGPT tab. 2. The paper calls text-davinci-003 the clearly better of the GPT models for translation out of English §3.2.
• Few-shot helps most when translating out of English: a single high-quality shot gives most of the gain. For into-English pairs, the few-shot gain is small §3.3.
• Lexical metrics (BLEU, chrF) fall for all GPT settings, while COMET scores do not §3.2 §3.3.
• Low-resource pairs: GPT does not beat WMT-best for IS and HA §3.4. Human evaluation puts GPT ahead of WMT-best on CS-EN, ZH-EN, EN-ZH and DE-FR, and behind on the two low-resource languages §4.
• Document level (DE-EN and EN-DE, WMT22) tab. 5:

• Document-level few-shot gives no clear gain over zero-shot document translation §3.5 tab. 6.
• Domain shift (German and Chinese, four WMT22 domains): GPT is strong on the conversational domain and on DE-EN overall; BLEU for ZH-EN and DE-EN falls on news and social text §3.6 tab. 7.
• Hybrid: Max-routing gives the highest COMET-22 in all 12 directions shown, with a maximum gain of 1.6 COMET-22 points on EN-UK. The threshold hybrid matches the upper bound on all pairs while calling GPT for about half of the instances §3.7 fig. 1.
• Released MS-Translator scores match the table values on the eight directions spot-checked in the released score file, to the one-decimal precision of the paper scores.txt.

Paper vs code

Official repository microsoft/gpt-MT (MIT, Microsoft Corporation), commit d64e21fbc0fa60f3e9ca70bfe4cddf1488177531. It holds data and scoring code, not the API calling code.

• Shots: released per direction under data-shots/ for RR, QR, QS, DR and DF . The repository does not include the script that builds the LaBSE-ranked pool, so the shot construction cannot be re-run from this code.
• Prompts: the README gives zero-shot and few-shot templates in the davinci format README.md. It shows no ChatGPT template, while the paper prints a separate ChatGPT zero-shot template app. C. Which template produced the released ChatGPT outputs is not stated.
• Doc-COMET defaults: the code's context length defaults to 4 sentences and stride 1 doc_score.py. The paper does not state the window used for its Doc-COMETkiwi column. The code also caps each input at 512 tokens and shrinks the context to fit doc_score.py doc_score.py.
• BLEU tokenization: the scoring script uses 13a by default, ja-mecab for Japanese targets and zh for Chinese targets [code: tools/evaluate.py#L82]. The paper does not state the tokenizer.
• Extra models: the repository ships ChatGPT and GPT-4 outputs . The paper evaluates three models and does not describe GPT-4.
• Hybrid outputs: the repository ships Max-routing and threshold outputs for text-davinci-003 . The threshold computation itself is not in the repository.
• Author notes in the LaTeX source: the source contains two unresolved notes about table entries: one says a DE-EN 5-shot QS entry was replaced by a 1-shot value, and one says the EN-ZH 5-shot RR and QS values do not match (LaTeX source, lines 441-442). Check any figure you reuse against the released outputs.

No discrepancies were found in the hyperparameters this card lists, other than those above.

Pitfalls

• API models are not pinned. The paper names text-davinci-002, text-davinci-003 and ChatGPT, and accessed them via the Azure OpenAI Service §2.3. No model snapshot or access date is recorded. Re-running these prompts on other models is a different experiment.
• Sampling settings are missing. Temperature, top-p, and maximum output length are not stated, and no generation script is released. Treat the reported outputs as the reference, not a recipe.
• The quality pool filter is ambiguous. "Longer than 50 tokens" does not say which tokenizer counts the tokens, and the order between the LaBSE top-1M cut and the length filter is not stated §3.1.
• More shots do not always help. More than 5 shots gave no gain in preliminary tests, and relevance-selected shots gave no gain over quality shots when translating from English §3.1 §3.3. Do not assume that more retrieved examples improve translation.
• Lexical scores mislead here. BLEU and chrF drop for GPT output while COMET and human judgements do not. A pipeline that gates on BLEU alone would likely reject good output §3.2 §4.
• Document-level realignment is a post-processing step. After generation, sentence alignment is restored by inserting line breaks and blank lines §3.5. Output that reorders, merges or drops sentences makes the document metric harder to read.
• Hybrid routing needs a reference-free estimator and a threshold. The threshold is the median COMETkiwi score of MS-Translator output §3.7, which must be estimated from earlier traffic. Whether it is set per direction is not stated. Max-routing needs both outputs, so it is an upper bound, not a deployable method.
• Cost. Document-level translation cut GPT requests from 1984 to 310 on DE-EN in the reported setting tab. 5. The hybrid also depends on running the primary system for every input.
• Not covered. The paper's multilingual reasoning experiment (§6) and its per-language analysis tables are outside this card.

Lineage

• Builds on: the GPT-3 in-context learning setup §1 §3.1; the transformer architecture §1; InstructGPT, cited as the basis of text-davinci-002 §2.3.
• Compared against: WMT best systems and Microsoft Translator §2.2 tab. 1. The paper also places itself against LLM translation studies that use GLM-130B, PaLM-540B and XGLM-7.5B §3.1.
• Metrics used: COMET-22 and COMETkiwi from the WMT22 metrics shared task §2.4.
• Superseded by: not stated in the paper.