Neural machine translation

Overview

Neural machine translation (NMT) trains one network that reads a source sentence and writes a target sentence. The usual setup is an encoder-decoder trained on parallel sentence pairs, scored with BLEU 1909.05088 §4.2 1909.05088 §5. The invariant is that the model only sees the source sentence. Any information the source leaves out, such as the speaker's gender in a language that marks it, is lost unless something adds it [arxiv:1909.05088 §1].

Large language models can also translate when asked in plain text. The GPT paper describes them as decoder-only and trained mostly on monolingual text 2302.09210 §1. They are not encoder-decoder NMT systems, but the paper measures them against the strongest NMT systems, so this page covers them as a variant. A separate use of a language model is as the judge of a translation 2302.14520 §1.

Ten cards on this wiki touch the concept. They cover different stages: how the text is split into units before training 1508.07909 §3.2, whether a transcript is needed when the input is speech 1612.01744 §1, what extra input the model gets 1909.05088 §4.1, how translations are scored after the fact 2106.11520 §3.2, how professional raters mark the errors in a translation 2104.14478 §3.3, how a GPT model is prompted to translate 2302.09210 §2.3, how a GPT model rates a translation 2302.14520 §2, how one large model is built to cover 202 languages 2207.04672 §8.2.4, how an LLM translation feeds a downstream classifier 2410.12985 §3, and how a robot application calls a hosted translation service 2412.05022 §6.

Variants

Subword units (byte pair encoding)

Split words into units learned from the training text, so the model has a fixed vocabulary and needs no dictionary for rare words. The method learns a list of merge operations: start from characters, repeatedly join the most frequent adjacent pair, and apply the same merges to new text [arxiv:1508.07909 §3.2]. The only hyperparameter is the number of merges 1508.07909 §3.2. The paper uses 59,500 merges for the separate-encoding system and 89,500 for the joint one 1508.07909 §4.1. Card: Neural Machine Translation of Rare Words with Subword Units.

Two ways to learn the merges are tested. Separate encodings for source and target are one option. A joint encoding learned over both sides keeps the segmentation of shared names consistent, and the authors credit it for names 1508.07909 §3.2 1508.07909 §5.2. The authors also tested character bigrams with a 50,000-word shortlist, which is a different segmentation and not BPE 1508.07909 §4.1.

End-to-end speech-to-text translation

Replace the usual chain of speech recognition followed by translation with one attention-based encoder-decoder that reads speech features and writes target text, with no source transcript used in training or decoding [arxiv:1612.01744 §1]. The speech encoder is a hierarchical bidirectional LSTM over 40 MFCCs plus frame energy, with two fully connected layers in front, and each of its upper layers reads every other output of the layer below 1612.01744 §2 1612.01744 §3. The speech attention adds a convolution over the previous step's attention weights, which the authors say keeps the model from translating the same part of the signal twice 1612.01744 §2 1612.01744 eq. 6. The experiments use synthetic French speech and English text from a small travel-phrase corpus, so the results are a proof of concept 1612.01744 §3 1612.01744 §5. Card: Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation.

Speaker-gender source tag

Add one tag token to the start of each English source sentence, with the speaker's gender as its value, and train the model on the tagged data. The baseline is the same model trained on untagged data 1909.05088 §4.1. The paper uses LSTM encoder-decoder systems from OpenNMT-py, BPE with 89,500 operations and 13 training epochs 1909.05088 §4.2. Card: Getting Gender Right in Neural Machine Translation.

The tag is a training-time change. The paper's female tag is shown as FEMALE, and the male tag is not given 1909.05088 §4.1. The speaker labels come from predicted MEP gender metadata, and the paper does not measure how often they are wrong 1909.05088 §3. The authors report the results below for 10 language pairs, with BLEU gains of at most about 1.4 points on the general test sets 1909.05088 tab. 2.

Evaluation: BARTScore

BARTScore scores a generated text by the log-likelihood that a pre-trained BART encoder-decoder assigns to it, given a source or a reference, so no human labels are needed 2106.11520 §1 2106.11520 §3.2. It has four directions: source to hypothesis (faithfulness), reference to hypothesis (precision), hypothesis to reference (recall), and their arithmetic mean (F) 2106.11520 §3.2. Card: BARTScore: Evaluating Generated Text as Text Generation.

The paper defines four variants: vanilla BART, BART fine-tuned on CNN/DailyMail, the same model fine-tuned further on paraphrases, and a prompt-averaged version 2106.11520 §3.3 2106.11520 §4.2.2. For machine translation the authors use one prompt, "such as", chosen on the WMT18 development set 2106.11520 §4.2.1. That selection is not in the released repository, and the card's Paper vs code section lists further differences from the paper, such as the default checkpoint.

Prompted GPT translation

Give a GPT model an instruction, a few source and reference example pairs and the new source sentence, and read its reply as the translation. The paper tests three OpenAI models, text-davinci-002, text-davinci-003 and ChatGPT, on 18 directions from the WMT22 test sets, with WMT21 for Icelandic and Hausa 2302.09210 §2.1 2302.09210 §2.3. It tries 0, 1 and 5 shots and reports no gain from more than 5 in preliminary runs 2302.09210 §3.1. When translating out of English, shots chosen by similarity to the input gave no gain over shots drawn from a quality-filtered pool 2302.09210 §3.3.

Translating a whole document in one request, with the output realigned to the source sentences, cut GPT requests on DE-EN from 1,984 to 310 at a window of 16 sentences. COMET-22 rose from 84.8 to 85.2 2302.09210 §3.5 2302.09210 tab. 5. Few-shot document translation gave no clear gain over zero-shot 2302.09210 §3.5 2302.09210 tab. 6. Card: How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.

A hybrid sends each sentence to Microsoft Translator first, and retranslates with GPT only when a reference-free quality estimate falls below the median of that system's scores 2302.09210 §3.7. The threshold version calls GPT for about half the sentences 2302.09210 §3.7 fig. 1. Max-routing keeps the better of the two outputs for each sentence. It needs both outputs, so it is an upper bound, and it reaches up to 1.6 COMET-22 points above the better single system on EN-UK 2302.09210 §3.7 fig. 1.

LLM as a translation judge

Prompt a GPT model to rate one translated segment with a fixed zero-shot instruction, with or without the human reference, then average the segment ratings per system 2302.14520 §2 2302.14520 §2.2. The paper tries four rating styles, each with and without a reference: direct assessment on 0 to 100, which the authors call their best-performing prompt; a 0 to 100 scale with anchor phrases; one to five stars; and five quality labels 2302.14520 §2.1. An answer that is not a valid value is asked again at a higher temperature, and the paper does not give the step size 2302.14520 §2.2 2302.14520 §4.5. Card: Large Language Models Are State-of-the-Art Evaluators of Translation Quality.

Error-annotated human evaluation (MQM)

Ask professional translators to mark every error span in a translated document, with a category and a severity, and turn the marks into a segment score. The paper's scale runs from 0 (perfect) to 25 (worst), with Minor errors weighing 1, Major errors weighing 5, and a whole-segment Non-translation error weighing 25 2104.14478 §3.3 2104.14478 tab. 1. Each annotator keeps at most five target errors per segment, the most severe ones 2104.14478 app. B. Three raters annotate each segment [arxiv:2104.14478 tab. 2]. Card: Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.

The paper compares this with the WMT20 crowd or researcher ratings and with a 0-to-6 scale rated by professionals (pSQM) and by crowd workers (cSQM) 2104.14478 §3.1 2104.14478 §3.2. It states the weights and the cap but releases no scoring script, so the aggregation must be written from the paper 2104.14478 §3.3. The public data release stores its average scores with the opposite sign, so check the sign before comparing (details on the card). The Major weight came from a resampling check over 1 to 10, whose procedure is summarised, not specified 2104.14478 §3.3.

Massively multilingual sparse model (NLLB-200)

Train one many-to-many model for 202 languages instead of one model per language pair. The final model is a Transformer encoder-decoder in which every fourth feed-forward block is a sparse mixture-of-experts layer with 128 experts, and each token goes to its top two 2207.04672 §8.2.2 2207.04672 §8.2.4. It has 54.5B parameters, while its compute per token is close to a 3.3B dense model 2207.04672 §8.2.4.

Two training choices carry most of the method. Expert output masking zeroes the output of the first and/or second expert for a random share of tokens, with rate 0.2 in the final model 2207.04672 §6.2.2 [arxiv:2207.04672 §8.2.4]. A four-phase curriculum groups language pairs into buckets that join training at fixed update counts, so that pairs that overfit early join late 2207.04672 §8.2.3. The paper's low-resource label is fewer than 1 million public sentence pairs 2207.04672 §2.1.

Training data mixes human-translated seed data, bitext mined from the web with LASER3 sentence encoders, and backtranslated monolingual text 2207.04672 §8.1.2 2207.04672 §5.3 2207.04672 §8.1.3. Mined pairs are filtered, for example by a LASER score below 1.06 2207.04672 §8.1.4. For serving, the authors distil the large model into dense students of 1.3B and 615M parameters 2207.04672 §8.6.3. Card: No Language Left Behind: Scaling Human-Centered Machine Translation.

Hosted translation service in an application

Send the text to an external translation API from inside an application and speak or display the reply. A case study on a Pepper humanoid does this for one foreign language, Danish, using DeepL, after the answer is taken from a knowledge base 2412.05022 §6. The paper says DeepL covers 27 target languages, but only Danish was built and tested 2412.05022 §6. The paper reports no translation quality measurement. Its only evaluation is qualitative feedback from workshops with public-service staff [arxiv:2412.05022 §7]. Customer utterances and answers go to the external service over the internet, and the paper names data protection as a concern for later without describing safeguards 2412.05022 §6 2412.05022 §7. Card: Get It Right: Improving Comprehensibility with Adaptable Speech Expression of a Humanoid Service Robot.

Translate-then-classify pipeline

Translate the input with a chat model, then run a classifier on the translation. In the paper's depression-severity study, GPT-3.5-turbo at temperature 0 translates English Reddit posts into Greek, and the same model then assigns each Greek post one of four severity levels 2410.12985 §3 §4 fig. 1. The translation prompt is not stated 2410.12985 §4. Card: Leveraging LLMs for Translating and Classifying Mental Health Data.

The reported macro F1 is 0.17 on the English posts and 0.16 on the Greek translations 2410.12985 tab. 2. The English run is the only baseline. The paper reports no translation quality metric, and its one check on the translations is a qualitative look at two examples 2410.12985 fig. 2. Each figure comes from a single run with one zero-shot prompt 2410.12985 §4. The Greek numbers therefore cannot separate translation effects from classifier effects.

When to use which

Implementation sketch

The steps are the common pipeline: segment, optionally tag, train, translate, score. For the GPT variants the translate step is a prompt and the score step is a judge prompt. Values are the authors' choices, not general rules.

# Learn merge operations on the training text  [arxiv:1508.07909 §3.2] [alg. 1]
# Merge count is a tuning parameter; the paper uses 59,500 or 89,500 [§4.1]
def learn_bpe(word_counts, num_merges):
    symbols = {w: split_chars_plus_end(w) for w in word_counts}
    merges = []
    for _ in range(num_merges):
        # Most frequent adjacent pair, weighted by word count [§3.2]
        pair = most_frequent_pair(symbols, word_counts)
        symbols = merge_pair(symbols, pair)
        merges.append(pair)
    return merges

# Attach speaker metadata to each source sentence  [arxiv:1909.05088 §3]
# Gender values are predictions from MEP metadata  [arxiv:1909.05088 §3]
def tag_pairs(pairs, speaker_gender):
    # Tag on the source side only; target is unchanged  [arxiv:1909.05088 §4.1]
    return [(gender_token[g] + " " + src, tgt)
            for (src, tgt), g in zip(pairs, speaker_gender)]

# Train two systems per language pair: baseline and tagged  [arxiv:1909.05088 §4.1]
# Joint or per-side BPE is not stated  [arxiv:1909.05088 §4.2]
def build_systems(pairs, tagged_pairs):
    merges = learn_bpe(pairs, operations=89500)
    baseline = train_nmt(apply_bpe(pairs, merges), epochs=13)
    tagged = train_nmt(apply_bpe(tagged_pairs, merges), epochs=13)
    return baseline, tagged

# Mean token log-likelihood of target given source  [arxiv:2106.11520 §3.2]
# Printed eq. 2 is a sum; the official code divides by length  [arxiv:2106.11520 eq. 2]
def bart_score(model, source, target):
    log_probs = token_log_probs(model, source, target)
    return sum(log_probs) / len(log_probs)

# F score: mean of precision (reference to hypothesis) and recall  [§3.2]
def bart_f_score(model, reference, hypothesis):
    precision = bart_score(model, reference, hypothesis)
    recall = bart_score(model, hypothesis, reference)
    return (precision + recall) / 2

# Few-shot prompt for a GPT model  [arxiv:2302.09210 §3.1] [app. C]
# Layout per the paper's template: instruction, numbered example pairs, then
# the input and an empty numbered slot. Instruction wording is not copied here.
def build_prompt(instruction, shots, source):
    lines = [instruction]
    for i, (src, ref) in enumerate(shots, start=1):
        lines.append(f"{i}. {src}")
        lines.append(f"{i}. {ref}")
    n = len(shots) + 1
    lines.append(f"{n}. {source}")
    lines.append(f"{n}.")  # the model fills this slot
    return "\n".join(lines)

# Rate one segment with the zero-shot direct-assessment prompt  [arxiv:2302.14520 §2.1]
# Retry an invalid answer at a higher temperature  [arxiv:2302.14520 §2.2] [§4.5]
def score_segment(source, hypothesis, reference, src_lang, tgt_lang):
    prompt = fill_da_template(
        source, hypothesis, reference, src_lang, tgt_lang
    )
    temperature = 0.0
    for _ in range(MAX_ATTEMPTS):
        reply = call_model(prompt, temperature)
        # Accept one integer in 0 to 100 only  [§2.2]
        score = parse_integer_in_range(reply, 0, 100)
        if score is not None:
            return score
        # The paper does not state the step size [§2.2]
        temperature = min(1.0, temperature + STEP)
    return None  # no valid answer after all attempts

# System score: mean of the valid segment scores  [arxiv:2302.14520 §2.2]
def system_score(segment_scores):
    valid = [s for s in segment_scores if s is not None]
    return sum(valid) / len(valid)

# MQM segment score, 0 (perfect) to 25 (worst)  [arxiv:2104.14478 §3.3]
# Weights and cap from the paper's table 1 and appendix B
def mqm_segment_score(annotator_errors):
    # Average over the three annotators of a segment  [arxiv:2104.14478 §3.3]
    per_annotator = [annotator_score(errs) for errs in annotator_errors]
    return sum(per_annotator) / len(per_annotator)

def annotator_score(errors):
    # Whole-segment Non-translation sets the score to 25  [tab. 1]
    if any(e.category == "Non-translation" for e in errors):
        return 25
    # Source errors do not count; keep the five most severe  [app. B]
    target = [e for e in errors if not e.is_source]
    kept = most_severe(target, limit=5)
    # Neutral 0, Minor 1, Major 5; Minor Fluency/Punctuation 0.1  [tab. 1]
    return sum(weight(e) for e in kept)

# Phased curriculum: each bucket joins at a fixed update  [arxiv:2207.04672 §8.2.3]
# Entry points of 0, 170k, 230k and 270k of 300k updates  [arxiv:2207.04672 §8.2.4]
def active_pairs(update, buckets, entry_updates):
    active = []
    for bucket, entry in zip(buckets, entry_updates):
        # A bucket is trained on once its entry update is reached  [§8.2.3]
        if update >= entry:
            active += bucket
    return active

Not stated in the papers: the male tag's token value, the decoding settings, the checkpoint rule and the BLEU tool 1909.05088 §4.2 1909.05088 §5. The BARTScore code reads a file of paraphrase weights that the repository does not contain, so the paraphrase variant cannot be reproduced from it alone (see the card). The GPT translation runs state no temperature, top-p or output length 2302.09210 §2.3, and the judge's retry step is not stated 2302.14520 §2.2.

Results

BPE results are author-reported on WMT 2015, with BLEU on newstest2015 and ensembles of eight models 1508.07909 §4 1508.07909 §4.2. The Δ column is computed here.

Source: 1508.07909 tab. 2 1508.07909 tab. 3. The paper's headline "1.1 and 1.3 BLEU" are ensemble gains over WDict, and they come from different systems: C2-50k on EN→DE and BPE-J90k on EN→RU. BPE-J90k gains 0.5 BLEU on EN→DE in the ensemble 1508.07909 tab. 2. On rare words, unigram F1 on EN→DE rises from 36.8% to 41.8% 1508.07909 §4.2 [arxiv:1508.07909 tab. 2].

Speaker-gender tag, author-reported. BLEU on the general 2K test set, baseline (EN) against tagged (EN-TAG). The Δ column is computed here from the two cited values. Stars mark significant results at p < 0.05, as the authors report 1909.05088 tab. 2.

On four EN-FR test sets the authors report gains of +0.66 to +1.25 BLEU, all starred significant, with the largest gains on the female-speaker and first-person sets 1909.05088 tab. 3 1909.05088 §5. The paper says part of the gain comes from word choice, not only from agreement, and gives one qualitative example 1909.05088 §5.

BARTScore on machine translation, author-reported. Kendall's tau, WMT19:

Source: 2106.11520 tab. 3 2106.11520 §4.3.1. The paper reports a 0.033 gain on de-en from the "such as" prompt 2106.11520 §4.3.1. The paper's headline is that BARTScore beats the top-scoring existing metrics in 16 of 22 test settings 2106.11520 §5.

GPT translation, author-reported. COMET-22 for the WMT-best system, Microsoft Translator and the best GPT row shown in the paper's tables, with its setting. The paper's main GPT model is text-davinci-003 unless noted [arxiv:2302.09210 §3.3]:

The paper reports that BLEU and chrF fall for all GPT settings, while COMET does not 2302.09210 §3.2 2302.09210 §3.3. Human evaluation puts GPT ahead of the WMT-best system on CS-EN, ZH-EN, EN-ZH and DE-FR, and behind on Icelandic and Hausa 2302.09210 §4.

LLM judge, author-reported. System-level pairwise accuracy on WMT22 MQM, for en-de, en-ru and zh-en together 2302.14520 §3.1:

Segment-level Kendall tau-b for GEMBA-GPT4-DA with reference is 0.36, 0.36 and 0.38 for the three pairs. COMET-22 and MetricX XXL range from 0.36 to 0.43 2302.14520 tab. 4.

MQM human evaluation, author-reported. Segment MQM score, where lower is better, with three professional raters per segment. Values from the paper's tables, for selected systems 2104.14478 tab. 3 [arxiv:2104.14478 tab. 4]:

Under MQM every human translation ranks above every MT system on both pairs 2104.14478 §4.1. The paper's English-German MT score is about 2.9 times the human one over all categories 2104.14478 tab. 5. The paper's metric study uses averages over the WMT20 metrics, shown below for system-level and segment-level Kendall and Pearson correlations 2104.14478 tab. 9:

The study rates only seven English-German systems for the metric part, so the system-level values are noisy 2104.14478 §4.6. Adding the human outputs to the scored systems lowers correlations, most of all under MQM 2104.14478 tab. 9.

NLLB-200, author-reported. Scores are spBLEU and chrF++, on FLORES-200 devtest unless noted. Averages are over all 40,602 directions 2207.04672 §8.3.2:

The paper's headline is +7.3 spBLEU over the nearest prior system on FLORES-101, a 44% relative gain 2207.04672 §8.3.1. The DeltaLM row is taken from the FLORES-101 leaderboard, not re-run by the authors 2207.04672 tab. 30. The curriculum ablation on the dev set gives 47.6 average chrF++ without a curriculum and 48.5 with the four phases 2207.04672 tab. 29.

End-to-end speech translation, author-reported. BLEU on the BTEC French speech set, with 7 references, for the held-out speaker Agnes and one training speaker (Michel). The pipeline column is Google Speech ASR plus SMT, with a word error rate of 23 to 26% 1612.01744 tab. 3:

The paper's text-to-text model on the same corpus reaches 60.3 BLEU on test1 with 7 references and an ensemble 1612.01744 tab. 2. The speech and text rows are not on the same test data, so compare them only with that caveat.

Contradictions

• Which pairs improve significantly. The text names French, Italian, Portuguese and Greek as significant improvements 1909.05088 §5. Table 2 stars only French, Italian and Danish as improvements, and Portuguese and Greek are not starred 1909.05088 tab. 2.
• Significance threshold. Table 2's caption gives p < 0.05. Table 3's caption gives p < 0.5 1909.05088 tab. 2 1909.05088 tab. 3. The 0.05 value is the one used here.
• Headline judge prompt. For some experiments the text calls direct assessment the main style and reports 89.8% for it 2302.14520 §4. The paper's own table gives GPT-4 Stars and GPT-4 Classes[noref] 91.2% 2302.14520 tab. 3, and the text does not explain why direct assessment is the one it reports.
• Training length. The paper says the final NLLB-200 model trains for 300k updates 2207.04672 §8.2.4. The released final script stops at 170k updates. The paper card reads the code and explains the difference 2207.04672 §8.2.4. Check which one a comparison used.
• Curriculum effect. Section 6.2.3 says a curriculum hurt the MoE model with expert output masking, in the ablation at 100k updates. The 300k ablation in table 29 shows the four-phase curriculum ahead of no curriculum 2207.04672 §6.2.3 2207.04672 tab. 29. The two settings differ, and the paper does not reconcile them in the sections read here.
• Text and table. The text gives 39.5 chrF++ for supervised xx-yy directions, and table 33 gives 39.7 2207.04672 §8.3.2 2207.04672 tab. 33.
• Rater spread. The text says English-German MQM raters fall within ±20% of the mean 2104.14478 §4.4. The paper's rater table gives Rater 3 at 1.23 of the average 2104.14478 tab. 7, which is outside that band. Both are reported; the table is the only per-rater figure.

Papers

• Neural Machine Translation of Rare Words with Subword Units: byte pair encoding for subword units, with WMT 2015 English-German and English-Russian results against a dictionary back-off.
• Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation: one attention-based encoder-decoder maps French speech straight to English text on synthetic BTEC speech, against a Google ASR plus SMT pipeline.
• Getting Gender Right in Neural Machine Translation: adds a speaker-gender tag to the English source of Europarl sentences and reports BLEU changes for 10 language pairs and four EN-FR test sets.
• BARTScore: Evaluating Generated Text as Text Generation: scores generated text by BART log-likelihood, with a machine-translation setting on WMT19.
• How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation: compares three OpenAI GPT models with WMT-best systems and Microsoft Translator on 18 directions, with shots, document windows and a hybrid.
• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation: professional MQM error annotation of WMT20 outputs, compared with WMT ratings and with the WMT20 metrics.
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality: GEMBA, zero-shot GPT prompts that rate each translated segment, compared with the WMT22 metrics.
• No Language Left Behind: Scaling Human-Centered Machine Translation: NLLB-200, a 54.5B sparse mixture-of-experts model for 202 languages, with mined and backtranslated training data and FLORES-200 evaluation.
• Leveraging LLMs for Translating and Classifying Mental Health Data: GPT-3.5-turbo translates English Reddit posts into Greek, then classifies depression severity in both languages, with no translation metric.
• Get It Right: Improving Comprehensibility with Adaptable Speech Expression of a Humanoid Service Robot: a Pepper robot app that restates answers in German easy language or translates them into Danish through hosted APIs, with no quantitative evaluation.