---
kind: concept
slug: neural-machine-translation
title: Neural machine translation
summary: >-
  Neural machine translation (NMT) maps a source sentence to a target sentence with one trained network. Covers BPE subwords, a speaker-gender tag, speech-to-text translation, BARTScore, GPT translation and judging, MQM annotation, NLLB-200 and an LLM translate-then-classify pipeline. Results are author-reported; gender-tag BLEU gains are under 1.5 points.
tags: [machine-translation, encoder-decoder, lstm, subword, speech-translation, evaluation, human-evaluation, error-annotation, speaker-attributes, gender-bias, large-language-models, llm-as-a-judge, low-resource-languages, mixture-of-experts, data-mining, text-classification]
papers: ["1508.07909", "1612.01744", "1909.05088", "2104.14478", "2106.11520", "2302.09210", "2302.14520", "2207.04672", "2410.12985", "2412.05022"]
related: [controllable-text-generation, algorithmic-fairness, recurrent-neural-network, long-short-term-memory, attention-mechanism, decoding-strategies, likelihood-based-text-evaluation, prompt-engineering, subword-tokenization, in-context-learning, llm-as-a-judge, llm-evaluation, human-evaluation, benchmark-validity, mixture-of-experts, dataset-curation, curriculum-learning, knowledge-distillation, multilingual-evaluation, text-simplification, text-classification]
status: draft
updated_at: 2026-11-01
---

## Overview

Neural machine translation (NMT) trains one network that reads a source
sentence and writes a target sentence. The usual setup is an encoder-decoder
trained on parallel sentence pairs, scored with BLEU [arxiv:1909.05088 §4.2]
[arxiv:1909.05088 §5]. The invariant is that the model only sees the source
sentence. Any information the source leaves out, such as the speaker's gender
in a language that marks it, is lost unless something adds it [arxiv:1909.05088
§1].

Large language models can also translate when asked in plain text. The GPT
paper describes them as decoder-only and trained mostly on monolingual text
[arxiv:2302.09210 §1]. They are not encoder-decoder NMT systems, but the paper
measures them against the strongest NMT systems, so this page covers them as a
variant. A separate use of a language model is as the judge of a translation
[arxiv:2302.14520 §1].

Ten cards on this wiki touch the concept. They cover different stages: how
the text is split into units before training [arxiv:1508.07909 §3.2], whether
a transcript is needed when the input is speech [arxiv:1612.01744 §1], what
extra input the model gets [arxiv:1909.05088 §4.1], how translations are
scored after the fact [arxiv:2106.11520 §3.2], how professional raters mark
the errors in a translation [arxiv:2104.14478 §3.3], how a GPT model is
prompted to translate [arxiv:2302.09210 §2.3], how a GPT model rates a
translation [arxiv:2302.14520 §2], how one large model is built to cover
202 languages [arxiv:2207.04672 §8.2.4], how an LLM translation feeds a
downstream classifier [arxiv:2410.12985 §3], and how a robot application calls
a hosted translation service [arxiv:2412.05022 §6].

## Variants

### Subword units (byte pair encoding)

Split words into units learned from the training text, so the model has a
fixed vocabulary and needs no dictionary for rare words. The method learns a
list of merge operations: start from characters, repeatedly join the most
frequent adjacent pair, and apply the same merges to new text [arxiv:1508.07909
§3.2]. The only hyperparameter is the number of merges [arxiv:1508.07909 §3.2].
The paper uses 59,500 merges for the separate-encoding system and 89,500 for
the joint one [arxiv:1508.07909 §4.1]. Card:
[Neural Machine Translation of Rare Words with Subword Units](/papers/1508.07909).

Two ways to learn the merges are tested. Separate encodings for source and
target are one option. A joint encoding learned over both sides keeps the
segmentation of shared names consistent, and the authors credit it for names
[arxiv:1508.07909 §3.2] [arxiv:1508.07909 §5.2]. The authors also tested
character bigrams with a 50,000-word shortlist, which is a different
segmentation and not BPE [arxiv:1508.07909 §4.1].

### End-to-end speech-to-text translation

Replace the usual chain of speech recognition followed by translation with one
attention-based encoder-decoder that reads speech features and writes target
text, with no source transcript used in training or decoding [arxiv:1612.01744
§1]. The speech encoder is a hierarchical bidirectional LSTM over 40 MFCCs plus
frame energy, with two fully connected layers in front, and each of its upper
layers reads every other output of the layer below [arxiv:1612.01744 §2]
[arxiv:1612.01744 §3]. The speech attention adds a convolution over the
previous step's attention weights, which the authors say keeps the model from
translating the same part of the signal twice [arxiv:1612.01744 §2]
[arxiv:1612.01744 eq. 6]. The experiments use synthetic French speech and
English text from a small travel-phrase corpus, so the results are a proof of
concept [arxiv:1612.01744 §3] [arxiv:1612.01744 §5]. Card:
[Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](/papers/1612.01744).

### Speaker-gender source tag

Add one tag token to the start of each English source sentence, with the
speaker's gender as its value, and train the model on the tagged data. The
baseline is the same model trained on untagged data [arxiv:1909.05088 §4.1].
The paper uses LSTM encoder-decoder systems from OpenNMT-py, BPE with 89,500
operations and 13 training epochs [arxiv:1909.05088 §4.2]. Card:
[Getting Gender Right in Neural Machine Translation](/papers/1909.05088).

The tag is a training-time change. The paper's female tag is shown as FEMALE,
and the male tag is not given [arxiv:1909.05088 §4.1]. The speaker labels
come from predicted MEP gender metadata, and the paper does not measure how
often they are wrong [arxiv:1909.05088 §3]. The authors report the results
below for 10 language pairs, with BLEU gains of at most about 1.4 points on the
general test sets [arxiv:1909.05088 tab. 2].

### Evaluation: BARTScore

BARTScore scores a generated text by the log-likelihood that a pre-trained
BART encoder-decoder assigns to it, given a source or a reference, so no human
labels are needed [arxiv:2106.11520 §1] [arxiv:2106.11520 §3.2]. It has four
directions: source to hypothesis (faithfulness), reference to hypothesis
(precision), hypothesis to reference (recall), and their arithmetic mean (F)
[arxiv:2106.11520 §3.2]. Card:
[BARTScore: Evaluating Generated Text as Text Generation](/papers/2106.11520).

The paper defines four variants: vanilla BART, BART fine-tuned on CNN/DailyMail,
the same model fine-tuned further on paraphrases, and a prompt-averaged
version [arxiv:2106.11520 §3.3] [arxiv:2106.11520 §4.2.2]. For machine
translation the authors use one prompt, "such as", chosen on the WMT18
development set [arxiv:2106.11520 §4.2.1]. That selection is not in the
released repository, and the card's Paper vs code section lists further
differences from the paper, such as the default checkpoint.

### Prompted GPT translation

Give a GPT model an instruction, a few source and reference example pairs and
the new source sentence, and read its reply as the translation. The paper
tests three OpenAI models, text-davinci-002, text-davinci-003 and ChatGPT, on
18 directions from the WMT22 test sets, with WMT21 for Icelandic and Hausa
[arxiv:2302.09210 §2.1] [arxiv:2302.09210 §2.3]. It tries 0, 1 and 5 shots and
reports no gain from more than 5 in preliminary runs [arxiv:2302.09210 §3.1].
When translating out of English, shots chosen by similarity to the input gave
no gain over shots drawn from a quality-filtered pool [arxiv:2302.09210 §3.3].

Translating a whole document in one request, with the output realigned to the
source sentences, cut GPT requests on DE-EN from 1,984 to 310 at a window of
16 sentences. COMET-22 rose from 84.8 to 85.2 [arxiv:2302.09210 §3.5]
[arxiv:2302.09210 tab. 5]. Few-shot document translation gave no clear gain
over zero-shot [arxiv:2302.09210 §3.5] [arxiv:2302.09210 tab. 6]. Card:
[How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](/papers/2302.09210).

A hybrid sends each sentence to Microsoft Translator first, and retranslates
with GPT only when a reference-free quality estimate falls below the median
of that system's scores [arxiv:2302.09210 §3.7]. The threshold version calls
GPT for about half the sentences [arxiv:2302.09210 §3.7] [fig. 1]. Max-routing
keeps the better of the two outputs for each sentence. It needs both outputs,
so it is an upper bound, and it reaches up to 1.6 COMET-22 points above the
better single system on EN-UK [arxiv:2302.09210 §3.7] [fig. 1].

### LLM as a translation judge

Prompt a GPT model to rate one translated segment with a fixed zero-shot
instruction, with or without the human reference, then average the segment
ratings per system [arxiv:2302.14520 §2] [arxiv:2302.14520 §2.2]. The paper
tries four rating styles, each with and without a reference: direct assessment
on 0 to 100, which the authors call their best-performing prompt; a 0 to 100
scale with anchor phrases; one to five stars; and five quality labels
[arxiv:2302.14520 §2.1]. An answer that is not a valid value is asked again at
a higher temperature, and the paper does not give the step size
[arxiv:2302.14520 §2.2] [arxiv:2302.14520 §4.5]. Card:
[Large Language Models Are State-of-the-Art Evaluators of Translation Quality](/papers/2302.14520).

### Error-annotated human evaluation (MQM)

Ask professional translators to mark every error span in a translated
document, with a category and a severity, and turn the marks into a segment
score. The paper's scale runs from 0 (perfect) to 25 (worst), with Minor
errors weighing 1, Major errors weighing 5, and a whole-segment Non-translation
error weighing 25 [arxiv:2104.14478 §3.3] [arxiv:2104.14478 tab. 1]. Each
annotator keeps at most five target errors per segment, the most severe ones
[arxiv:2104.14478 app. B]. Three raters annotate each segment [arxiv:2104.14478
tab. 2]. Card: [Experts, Errors, and Context: A Large-Scale Study of Human
Evaluation for Machine Translation](/papers/2104.14478).

The paper compares this with the WMT20 crowd or researcher ratings and with a
0-to-6 scale rated by professionals (pSQM) and by crowd workers (cSQM)
[arxiv:2104.14478 §3.1] [arxiv:2104.14478 §3.2]. It states the weights and the
cap but releases no scoring script, so the aggregation must be written from
the paper [arxiv:2104.14478 §3.3]. The public data release stores its average
scores with the opposite sign, so check the sign before comparing (details on
the card). The Major weight came from a resampling check over 1 to 10, whose
procedure is summarised, not specified [arxiv:2104.14478 §3.3].

### Massively multilingual sparse model (NLLB-200)

Train one many-to-many model for 202 languages instead of one model per
language pair. The final model is a Transformer encoder-decoder in which every
fourth feed-forward block is a sparse mixture-of-experts layer with 128
experts, and each token goes to its top two [arxiv:2207.04672 §8.2.2]
[arxiv:2207.04672 §8.2.4]. It has 54.5B parameters, while its compute per
token is close to a 3.3B dense model [arxiv:2207.04672 §8.2.4].

Two training choices carry most of the method. Expert output masking zeroes
the output of the first and/or second expert for a random share of tokens,
with rate 0.2 in the final model [arxiv:2207.04672 §6.2.2] [arxiv:2207.04672
§8.2.4]. A four-phase curriculum groups language pairs into buckets that join
training at fixed update counts, so that pairs that overfit early join late
[arxiv:2207.04672 §8.2.3]. The paper's low-resource label is fewer than 1
million public sentence pairs [arxiv:2207.04672 §2.1].

Training data mixes human-translated seed data, bitext mined from the web with
LASER3 sentence encoders, and backtranslated monolingual text
[arxiv:2207.04672 §8.1.2] [arxiv:2207.04672 §5.3] [arxiv:2207.04672 §8.1.3].
Mined pairs are filtered, for example by a LASER score below 1.06
[arxiv:2207.04672 §8.1.4]. For serving, the authors distil the large model
into dense students of 1.3B and 615M parameters [arxiv:2207.04672 §8.6.3].
Card: [No Language Left Behind: Scaling Human-Centered Machine Translation](/papers/2207.04672).

### Hosted translation service in an application

Send the text to an external translation API from inside an application and
speak or display the reply. A case study on a Pepper humanoid does this for one
foreign language, Danish, using DeepL, after the answer is taken from a
knowledge base [arxiv:2412.05022 §6]. The paper says DeepL covers 27 target
languages, but only Danish was built and tested [arxiv:2412.05022 §6]. The
paper reports no translation quality measurement. Its only evaluation is
qualitative feedback from workshops with public-service staff [arxiv:2412.05022
§7]. Customer utterances and answers go to the external service over the
internet, and the paper names data protection as a concern for later without
describing safeguards [arxiv:2412.05022 §6] [arxiv:2412.05022 §7]. Card:
[Get It Right: Improving Comprehensibility with Adaptable Speech Expression of a Humanoid Service Robot](/papers/2412.05022).

### Translate-then-classify pipeline

Translate the input with a chat model, then run a classifier on the
translation. In the paper's depression-severity study, GPT-3.5-turbo at
temperature 0 translates English Reddit posts into Greek, and the same model
then assigns each Greek post one of four severity levels [arxiv:2410.12985 §3]
[§4] [fig. 1]. The translation prompt is not stated [arxiv:2410.12985 §4].
Card: [Leveraging LLMs for Translating and Classifying Mental Health Data](/papers/2410.12985).

The reported macro F1 is 0.17 on the English posts and 0.16 on the Greek
translations [arxiv:2410.12985 tab. 2]. The English run is the only baseline.
The paper reports no translation quality metric, and its one check on the
translations is a qualitative look at two examples [arxiv:2410.12985 fig. 2].
Each figure comes from a single run with one zero-shot prompt [arxiv:2410.12985 §4].
The Greek numbers therefore cannot separate translation effects from
classifier effects.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Open vocabulary, rare words and compounds, and no back-off dictionary wanted | Subword units (BPE), joint if source and target share names | BPE needs no shortlist, and its only setting is the merge count [arxiv:1508.07909 §3.2]. On EN→RU, joint BPE ties C2-50k for the best ensemble score in the paper [arxiv:1508.07909 tab. 3] |
| You can retrain the NMT model and have speaker gender labels for the training sentences | Tag on the source side, as in the card | The tag is one token, and the paper's result is a change of this size [arxiv:1909.05088 §4.1] [arxiv:1909.05088 tab. 2] |
| Your labels are predicted, not verified | Measure label accuracy before training | The paper does not report label error, and wrong labels would tag sentences with the wrong gender [arxiv:1909.05088 §3] |
| Your target language does not show the feature, or your data are heavily skewed | Expect small or no gain | The paper finds the largest gains on female-speaker sentences in a corpus that is 67.39% male [arxiv:1909.05088 §3.1] [arxiv:1909.05088 §5] |
| You cannot retrain | BPE and the tag are not covered | Both change training. The tag paper trains every system from scratch and does not test adding the tag to a finished model [arxiv:1909.05088 §4.1] |
| You need automatic MT scores and have no human ratings | BARTScore, vanilla BART or the one-prompt MT setting | It uses the pre-trained model's output layer and adds no parameters [arxiv:2106.11520 §1]. Its average Kendall's tau on WMT19 is close to the supervised COMET [arxiv:2106.11520 tab. 3] |
| The score must reflect factuality | Vanilla or CNN BARTScore, not the prompt or paraphrase variants | Prompts did not help factuality, and paraphrase fine-tuning lowered it on Rank19 [arxiv:2106.11520 §4.3.2] |
| Widely spoken languages, access to a large GPT model, and per-request cost is acceptable | Prompted GPT, zero-shot or few-shot, with document windows if the text is long | Against the WMT-best system, the best GPT row is at or above on DE-EN, JA-EN, ZH-EN and EN-ZH, and up to 3.4 COMET-22 points below on the other directions [arxiv:2302.09210 tab. 3] [arxiv:2302.09210 tab. 4] |
| Icelandic or Hausa is in the language pair | A dedicated MT system, such as the WMT-best one | GPT did not beat WMT-best for Icelandic or Hausa in either direction. The EN→IS gap is 8.9 COMET-22 points [arxiv:2302.09210 §3.4] [arxiv:2302.09210 tab. 4] |
| You need a quality score for a whole test set, have reference translations, and can call a GPT-4-class model | LLM judge, direct assessment with the reference | 89.8% system-level pairwise accuracy on WMT22 MQM, above MetricX XXL at 85.0% [arxiv:2302.14520 tab. 2]. Babbage, Curie, GPT-2 and Ada gave weak or no usable scores [arxiv:2302.14520 §4.3] |
| You need to rank single segments, or your language pair is not one of the three tested | A trained metric such as COMET-22 or MetricX XXL, plus a human check | Segment-level Kendall tau-b for the judge is 0.36 to 0.38, against 0.36 to 0.43 for COMET-22 and MetricX XXL [arxiv:2302.14520 tab. 4]. The paper tests en-de, en-ru and zh-en only [arxiv:2302.14520 §3.1] |
| You need to know which error types hurt a system, and can pay professional raters | MQM error annotation, three raters per segment | Errors carry a category and a severity, and the paper finds the best systems mostly penalised for mistranslation and Non-translation [arxiv:2104.14478 §4.2] [arxiv:2104.14478 tab. 5]. The paper's rough cost estimate is about 1 USD per segment with three annotators, and it is not a measured cost [arxiv:2104.14478 app. A] |
| You compare human translations with MT, or check a metric against human judgments | MQM scores, with the protocol difference noted | Under MQM every human translation ranks above every MT system on both pairs. WMT's ranking puts Human-P last of ten on English-German [arxiv:2104.14478 §4.1] [arxiv:2104.14478 tab. 3]. Most metrics correlate better with MQM than with WMT [arxiv:2104.14478 §4.6] |
| You plan a rating budget for system ranking | Budget from the paper's estimates, then check it | The paper's simulated counts are 951 ratings per system for English-German and 3720 for Chinese-English, for expected Kendall tau 0.9. They are estimates from a Gaussian bootstrap, not new annotation [arxiv:2104.14478 tab. 8] [arxiv:2104.14478 §4.5]. For a fixed budget of 900, spreading ratings over more documents gave more accurate system scores [arxiv:2104.14478 §4.5] |
| Many low-resource pairs, one model wanted, and a 54.5B-parameter model can be served | Sparse multilingual model, as in NLLB-200 | One network covers 202 languages. Most of its 40,602 directions are zero-shot, and those score below the supervised pairs, 35.4 against 39.7 chrF++ on xx-yy [arxiv:2207.04672 §8.3.2] [arxiv:2207.04672 tab. 33] |
| Speech input, transcripts for the source language are costly to collect, and a proof of concept is enough | End-to-end speech-to-text, as in the card | The paper's motivation is that transcripts are costly for under-resourced languages [arxiv:1612.01744 §1]. On the held-out speaker's test1 set with 7 references, the 5-model ensemble trails the Google ASR plus SMT pipeline, 48.8 against 52.7 BLEU [arxiv:1612.01744 tab. 3]. The synthetic data limits what this shows [arxiv:1612.01744 §5] |
| Serving memory or latency limits the large model | A dense student distilled from it | The paper's 1.3B and 615M students gain 0.5 and 0.3 chrF++ over their dense baselines on average [arxiv:2207.04672 §8.6.3]. The large model needs all experts in memory [arxiv:2207.04672 §8.2.4] |
| You mine web bitext for a low-resource pair | The paper's mining filters as a starting point | The paper removes mined pairs with a LASER score below 1.06 and keeps a direction only if its xsim error rate is under 5 [arxiv:2207.04672 §8.1.4] [arxiv:2207.04672 §8.1.2] |
| You translate text only to feed a downstream classifier, and you have no translation metric for the new language | Measure translation quality on a sample before trusting the classifier's scores in that language | The paper has no translation metric, and its Greek F1 of 0.16 mixes translation and classifier effects [arxiv:2410.12985 §4] [tab. 2] |
| You add translation to an application through a hosted API and have no quality measurements of your own | Hosted NMT service for the target language, with your own quality check before users rely on it | The case study measures no translation quality and tests one language pair [arxiv:2412.05022 §6] [arxiv:2412.05022 §7]. The paper says the services "are not proprietary" without defining the term, so check the terms of the service first [arxiv:2412.05022 §6] |
| You compare against a commercial system on low-resource pairs | Report with the paper's caveat | NLLB-200 averages 35.8 spBLEU against 34.1 for Google Translate over 102 low-resource directions [arxiv:2207.04672 tab. 34]. The authors say commercial systems are proprietary and few details are public [arxiv:2207.04672 §8.3.2] |

## Implementation sketch

The steps are the common pipeline: segment, optionally tag, train, translate,
score. For the GPT variants the translate step is a prompt and the score step
is a judge prompt. Values are the authors' choices, not general rules.

```pseudocode
# Learn merge operations on the training text  [arxiv:1508.07909 §3.2] [alg. 1]
# Merge count is a tuning parameter; the paper uses 59,500 or 89,500 [§4.1]
def learn_bpe(word_counts, num_merges):
    symbols = {w: split_chars_plus_end(w) for w in word_counts}
    merges = []
    for _ in range(num_merges):
        # Most frequent adjacent pair, weighted by word count [§3.2]
        pair = most_frequent_pair(symbols, word_counts)
        symbols = merge_pair(symbols, pair)
        merges.append(pair)
    return merges
```

```pseudocode
# Attach speaker metadata to each source sentence  [arxiv:1909.05088 §3]
# Gender values are predictions from MEP metadata  [arxiv:1909.05088 §3]
def tag_pairs(pairs, speaker_gender):
    # Tag on the source side only; target is unchanged  [arxiv:1909.05088 §4.1]
    return [(gender_token[g] + " " + src, tgt)
            for (src, tgt), g in zip(pairs, speaker_gender)]
```

```pseudocode
# Train two systems per language pair: baseline and tagged  [arxiv:1909.05088 §4.1]
# Joint or per-side BPE is not stated  [arxiv:1909.05088 §4.2]
def build_systems(pairs, tagged_pairs):
    merges = learn_bpe(pairs, operations=89500)
    baseline = train_nmt(apply_bpe(pairs, merges), epochs=13)
    tagged = train_nmt(apply_bpe(tagged_pairs, merges), epochs=13)
    return baseline, tagged
```

```pseudocode
# Mean token log-likelihood of target given source  [arxiv:2106.11520 §3.2]
# Printed eq. 2 is a sum; the official code divides by length  [arxiv:2106.11520 eq. 2]
def bart_score(model, source, target):
    log_probs = token_log_probs(model, source, target)
    return sum(log_probs) / len(log_probs)

# F score: mean of precision (reference to hypothesis) and recall  [§3.2]
def bart_f_score(model, reference, hypothesis):
    precision = bart_score(model, reference, hypothesis)
    recall = bart_score(model, hypothesis, reference)
    return (precision + recall) / 2
```

```pseudocode
# Few-shot prompt for a GPT model  [arxiv:2302.09210 §3.1] [app. C]
# Layout per the paper's template: instruction, numbered example pairs, then
# the input and an empty numbered slot. Instruction wording is not copied here.
def build_prompt(instruction, shots, source):
    lines = [instruction]
    for i, (src, ref) in enumerate(shots, start=1):
        lines.append(f"{i}. {src}")
        lines.append(f"{i}. {ref}")
    n = len(shots) + 1
    lines.append(f"{n}. {source}")
    lines.append(f"{n}.")  # the model fills this slot
    return "\n".join(lines)
```

```pseudocode
# Rate one segment with the zero-shot direct-assessment prompt  [arxiv:2302.14520 §2.1]
# Retry an invalid answer at a higher temperature  [arxiv:2302.14520 §2.2] [§4.5]
def score_segment(source, hypothesis, reference, src_lang, tgt_lang):
    prompt = fill_da_template(
        source, hypothesis, reference, src_lang, tgt_lang
    )
    temperature = 0.0
    for _ in range(MAX_ATTEMPTS):
        reply = call_model(prompt, temperature)
        # Accept one integer in 0 to 100 only  [§2.2]
        score = parse_integer_in_range(reply, 0, 100)
        if score is not None:
            return score
        # The paper does not state the step size [§2.2]
        temperature = min(1.0, temperature + STEP)
    return None  # no valid answer after all attempts

# System score: mean of the valid segment scores  [arxiv:2302.14520 §2.2]
def system_score(segment_scores):
    valid = [s for s in segment_scores if s is not None]
    return sum(valid) / len(valid)
```

```pseudocode
# MQM segment score, 0 (perfect) to 25 (worst)  [arxiv:2104.14478 §3.3]
# Weights and cap from the paper's table 1 and appendix B
def mqm_segment_score(annotator_errors):
    # Average over the three annotators of a segment  [arxiv:2104.14478 §3.3]
    per_annotator = [annotator_score(errs) for errs in annotator_errors]
    return sum(per_annotator) / len(per_annotator)

def annotator_score(errors):
    # Whole-segment Non-translation sets the score to 25  [tab. 1]
    if any(e.category == "Non-translation" for e in errors):
        return 25
    # Source errors do not count; keep the five most severe  [app. B]
    target = [e for e in errors if not e.is_source]
    kept = most_severe(target, limit=5)
    # Neutral 0, Minor 1, Major 5; Minor Fluency/Punctuation 0.1  [tab. 1]
    return sum(weight(e) for e in kept)
```

```pseudocode
# Phased curriculum: each bucket joins at a fixed update  [arxiv:2207.04672 §8.2.3]
# Entry points of 0, 170k, 230k and 270k of 300k updates  [arxiv:2207.04672 §8.2.4]
def active_pairs(update, buckets, entry_updates):
    active = []
    for bucket, entry in zip(buckets, entry_updates):
        # A bucket is trained on once its entry update is reached  [§8.2.3]
        if update >= entry:
            active += bucket
    return active
```

Not stated in the papers: the male tag's token value, the decoding settings,
the checkpoint rule and the BLEU tool [arxiv:1909.05088 §4.2] [arxiv:1909.05088 §5].
The BARTScore code reads a file of paraphrase weights that the repository does
not contain, so the paraphrase variant cannot be reproduced from it alone
(see the card). The GPT translation runs state no temperature, top-p or output
length [arxiv:2302.09210 §2.3], and the judge's retry step is not stated
[arxiv:2302.14520 §2.2].

## Results

BPE results are author-reported on WMT 2015, with BLEU on newstest2015 and
ensembles of eight models [arxiv:1508.07909 §4] [arxiv:1508.07909 §4.2]. The
Δ column is computed here.

| System | EN→DE single | EN→DE ens-8 | EN→RU single | EN→RU ens-8 |
|---|---|---|---|---|
| WDict (word + dictionary) | 22.0 | 24.2 | 19.1 | 22.8 |
| C2-50k (character bigram) | 22.8 | 25.3 | 20.9 | 24.1 |
| BPE-60k | 21.5 | 24.5 | 20.5 | 23.6 |
| BPE-J90k (joint BPE) | 22.8 | 24.7 | 20.4 | 24.1 |

Source: [arxiv:1508.07909 tab. 2] [arxiv:1508.07909 tab. 3]. The paper's
headline "1.1 and 1.3 BLEU" are ensemble gains over WDict, and they come from
different systems: C2-50k on EN→DE and BPE-J90k on EN→RU. BPE-J90k gains 0.5
BLEU on EN→DE in the ensemble [arxiv:1508.07909 tab. 2]. On rare words, unigram
F1 on EN→DE rises from 36.8% to 41.8% [arxiv:1508.07909 §4.2] [arxiv:1508.07909
tab. 2].

Speaker-gender tag, author-reported. BLEU on the general 2K test set, baseline
(EN) against tagged (EN-TAG). The Δ column is computed here from the two cited
values. Stars mark significant results at p < 0.05, as the authors report
[arxiv:1909.05088 tab. 2].

| Language | EN | EN-TAG | Δ (derived) | Significant |
|---|---|---|---|---|
| FR | 37.82 | 39.26 | +1.44 | yes |
| IT | 31.46 | 31.75 | +0.29 | yes |
| DA | 36.69 | 37.00 | +0.31 | yes |
| FI | 21.82 | 21.35 | −0.47 | yes (decrease) |
| ES | 42.47 | 42.28 | −0.19 | no |
| DE | 28.28 | 28.05 | −0.23 | no |
| SV | 35.42 | 35.19 | −0.23 | no |
| NL | 28.35 | 28.22 | −0.13 | no |
| EL | 31.38 | 31.54 | +0.16 | not starred |
| PT | 36.11 | 36.33 | +0.22 | not starred |

On four EN-FR test sets the authors report gains of +0.66 to +1.25 BLEU, all
starred significant, with the largest gains on the female-speaker and
first-person sets [arxiv:1909.05088 tab. 3] [arxiv:1909.05088 §5]. The paper
says part of the gain comes from word choice, not only from agreement, and
gives one qualitative example [arxiv:1909.05088 §5].

BARTScore on machine translation, author-reported. Kendall's tau, WMT19:

| Setting | BARTScore result | Strongest comparison |
|---|---|---|
| Average over 7 language pairs | BARTScore+CNN+Para+Prompt 0.337 | COMET 0.339 (supervised); PRISM 0.326 (best other unsupervised) |
| de-en | BARTScore+CNN+Para+Prompt 0.238 | COMET 0.219; BLEURT 0.174 |

Source: [arxiv:2106.11520 tab. 3] [arxiv:2106.11520 §4.3.1]. The paper
reports a 0.033 gain on de-en from the "such as" prompt
[arxiv:2106.11520 §4.3.1]. The paper's headline is that BARTScore beats the
top-scoring existing metrics in 16 of 22 test settings [arxiv:2106.11520 §5].

GPT translation, author-reported. COMET-22 for the WMT-best system, Microsoft
Translator and the best GPT row shown in the paper's tables, with its setting.
The paper's main GPT model is text-davinci-003 unless noted [arxiv:2302.09210
§3.3]:

| Direction | WMT best | MS Translator | Best GPT (setting) | Source |
|---|---|---|---|---|
| DE-EN | 85.0 | 84.7 | 85.4 (5-shot, QR) | [arxiv:2302.09210 tab. 3] |
| EN-DE | 87.2 | 86.8 | 86.5 (5-shot, RR) | [arxiv:2302.09210 tab. 3] |
| ZH-EN | 81.0 | 80.4 | 81.6 (zero-shot) | [arxiv:2302.09210 tab. 3] |
| EN-IS | 86.8 | 84.3 | 77.9 (5-shot, RR) | [arxiv:2302.09210 tab. 4] |
| EN-HA | 79.8 | 72.5 | 74.1 (1-shot, QR) | [arxiv:2302.09210 tab. 4] |

The paper reports that BLEU and chrF fall for all GPT settings, while COMET
does not [arxiv:2302.09210 §3.2] [arxiv:2302.09210 §3.3]. Human evaluation
puts GPT ahead of the WMT-best system on CS-EN, ZH-EN, EN-ZH and DE-FR, and
behind on Icelandic and Hausa [arxiv:2302.09210 §4].

LLM judge, author-reported. System-level pairwise accuracy on WMT22 MQM, for
en-de, en-ru and zh-en together [arxiv:2302.14520 §3.1]:

| Metric | System-level accuracy | Source |
|---|---|---|
| GEMBA-GPT4-DA, with reference | 89.8% | [arxiv:2302.14520 tab. 2] |
| GEMBA-GPT4-DA, no reference | 87.6% | [arxiv:2302.14520 tab. 2] |
| MetricX XXL (non-public) | 85.0% | [arxiv:2302.14520 tab. 2] |
| COMETKiwi, no reference | 78.8% | [arxiv:2302.14520 tab. 2] |

Segment-level Kendall tau-b for GEMBA-GPT4-DA with reference is 0.36, 0.36 and
0.38 for the three pairs. COMET-22 and MetricX XXL range from 0.36 to 0.43
[arxiv:2302.14520 tab. 4].

MQM human evaluation, author-reported. Segment MQM score, where lower is
better, with three professional raters per segment. Values from the paper's
tables, for selected systems [arxiv:2104.14478 tab. 3] [arxiv:2104.14478 tab.
4]:

| System | EN-DE MQM | ZH-EN MQM |
|---|---|---|
| Human-B | 0.75 | 3.62 |
| Human-A | 0.91 | 3.43 |
| Human-P (paraphrased reference), EN-DE only | 1.41 | not in table |
| Best MT system in the table | 2.02 (Tohoku-AIP-NTT) | 5.03 (VolcTrans) |
| Worst MT system in the table | 2.99 (Online-A) | 5.85 (Online-B) |

Under MQM every human translation ranks above every MT system on both pairs
[arxiv:2104.14478 §4.1]. The paper's English-German MT score is about 2.9
times the human one over all categories [arxiv:2104.14478 tab. 5]. The
paper's metric study uses averages over the WMT20 metrics, shown below for
system-level and segment-level Kendall and Pearson correlations
[arxiv:2104.14478 tab. 9]:

| Average correlation | EN-DE WMT | EN-DE MQM | ZH-EN WMT | ZH-EN MQM |
|---|---|---|---|---|
| Pearson, system-level | 0.539 | 0.883 | 0.318 | 0.551 |
| Kendall, system-level | 0.436 | 0.637 | 0.309 | 0.443 |
| Kendall, segment-level | 0.170 | 0.228 | 0.159 | 0.298 |

The study rates only seven English-German systems for the metric part, so the
system-level values are noisy [arxiv:2104.14478 §4.6]. Adding the human
outputs to the scored systems lowers correlations, most of all under MQM
[arxiv:2104.14478 tab. 9].

NLLB-200, author-reported. Scores are spBLEU and chrF++, on FLORES-200 devtest
unless noted. Averages are over all 40,602 directions [arxiv:2207.04672 §8.3.2]:

| Setting | spBLEU | chrF++ | Source |
|---|---|---|---|
| Average, all directions | 17.5 | 35.7 | [arxiv:2207.04672 tab. 33] |
| Average, zero-shot xx-yy | 17.2 | 35.4 | [arxiv:2207.04672 tab. 33] |
| Average, supervised xx-yy | 20.3 | 39.7 | [arxiv:2207.04672 tab. 33] |
| FLORES-101, 101 languages: NLLB-200 | 24.0 | 41.7 | [arxiv:2207.04672 tab. 30] |
| FLORES-101, 101 languages: DeltaLM | 16.7 | not stated | [arxiv:2207.04672 tab. 30] |

The paper's headline is +7.3 spBLEU over the nearest prior system on FLORES-101,
a 44% relative gain [arxiv:2207.04672 §8.3.1]. The DeltaLM row is taken from
the FLORES-101 leaderboard, not re-run by the authors [arxiv:2207.04672 tab. 30].
The curriculum ablation on the dev set gives 47.6
average chrF++ without a curriculum and 48.5 with the four phases
[arxiv:2207.04672 tab. 29].

End-to-end speech translation, author-reported. BLEU on the BTEC French speech
set, with 7 references, for the held-out speaker Agnes and one training speaker
(Michel). The pipeline column is Google Speech ASR plus SMT, with a word error
rate of 23 to 26% [arxiv:1612.01744 tab. 3]:

| Speaker | Greedy | +Beam search | +LM | +Ensemble | Pipeline baseline |
|---|---|---|---|---|---|
| Agnes (held out) | 37.6 | 40.1 | 41.8 | 48.8 | 52.7 |
| Michel (in training) | 40.6 | 43.1 | 44.4 | 50.4 | 51.9 |

The paper's text-to-text model on the same corpus reaches 60.3 BLEU on test1
with 7 references and an ensemble [arxiv:1612.01744 tab. 2]. The speech and
text rows are not on the same test data, so compare them only with that caveat.

## Contradictions

- **Which pairs improve significantly.** The text names French, Italian,
  Portuguese and Greek as significant improvements [arxiv:1909.05088 §5]. Table
  2 stars only French, Italian and Danish as improvements, and Portuguese and
  Greek are not starred [arxiv:1909.05088 tab. 2].
- **Significance threshold.** Table 2's caption gives p < 0.05. Table 3's
  caption gives p < 0.5 [arxiv:1909.05088 tab. 2] [arxiv:1909.05088 tab. 3].
  The 0.05 value is the one used here.
- **Headline judge prompt.** For some experiments the text calls direct
  assessment the main style and reports 89.8% for it [arxiv:2302.14520 §4].
  The paper's own table gives GPT-4 Stars and GPT-4 Classes[noref] 91.2%
  [arxiv:2302.14520 tab. 3], and the text does not explain why direct
  assessment is the one it reports.
- **Training length.** The paper says the final NLLB-200 model trains for 300k
  updates [arxiv:2207.04672 §8.2.4]. The released final script stops at 170k
  updates. The [paper card](/papers/2207.04672) reads the code and explains
  the difference [arxiv:2207.04672 §8.2.4]. Check which one a comparison used.
- **Curriculum effect.** Section 6.2.3 says a curriculum hurt the MoE model
  with expert output masking, in the ablation at 100k updates. The 300k
  ablation in table 29 shows the four-phase curriculum ahead of no curriculum
  [arxiv:2207.04672 §6.2.3] [arxiv:2207.04672 tab. 29]. The two settings
  differ, and the paper does not reconcile them in the sections read here.
- **Text and table.** The text gives 39.5 chrF++ for supervised xx-yy
  directions, and table 33 gives 39.7 [arxiv:2207.04672 §8.3.2]
  [arxiv:2207.04672 tab. 33].
- **Rater spread.** The text says English-German MQM raters fall within
  ±20% of the mean [arxiv:2104.14478 §4.4]. The paper's rater table gives
  Rater 3 at 1.23 of the average [arxiv:2104.14478 tab. 7], which is outside
  that band. Both are reported; the table is the only per-rater figure.

## Papers

- [Neural Machine Translation of Rare Words with Subword Units](/papers/1508.07909): byte pair encoding for subword units, with WMT 2015 English-German and English-Russian results against a dictionary back-off.
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](/papers/1612.01744): one attention-based encoder-decoder maps French speech straight to English text on synthetic BTEC speech, against a Google ASR plus SMT pipeline.
- [Getting Gender Right in Neural Machine Translation](/papers/1909.05088): adds a speaker-gender tag to the English source of Europarl sentences and reports BLEU changes for 10 language pairs and four EN-FR test sets.
- [BARTScore: Evaluating Generated Text as Text Generation](/papers/2106.11520): scores generated text by BART log-likelihood, with a machine-translation setting on WMT19.
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](/papers/2302.09210): compares three OpenAI GPT models with WMT-best systems and Microsoft Translator on 18 directions, with shots, document windows and a hybrid.
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](/papers/2104.14478): professional MQM error annotation of WMT20 outputs, compared with WMT ratings and with the WMT20 metrics.
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](/papers/2302.14520): GEMBA, zero-shot GPT prompts that rate each translated segment, compared with the WMT22 metrics.
- [No Language Left Behind: Scaling Human-Centered Machine Translation](/papers/2207.04672): NLLB-200, a 54.5B sparse mixture-of-experts model for 202 languages, with mined and backtranslated training data and FLORES-200 evaluation.
- [Leveraging LLMs for Translating and Classifying Mental Health Data](/papers/2410.12985): GPT-3.5-turbo translates English Reddit posts into Greek, then classifies depression severity in both languages, with no translation metric.
- [Get It Right: Improving Comprehensibility with Adaptable Speech Expression of a Humanoid Service Robot](/papers/2412.05022): a Pepper robot app that restates answers in German easy language or translates them into Danish through hosted APIs, with no quantitative evaluation.