Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

In plain words

Machine translation is usually judged by people, but quick ratings of single sentences can miss the mistakes that matter, and small differences between good systems get lost in rater noise. This paper asks professional translators to read each translated passage in its document, mark every error they find, and label each one by type and by how serious it is. Those marks are then turned into a score.

The authors apply this to the top systems from a 2020 shared task. Under the error-based score, the order of the systems changes a lot from the original ranking. Professional human translations come out clearly ahead of the machine ones. Several automatic metrics agree better with the error-based score than with the original crowd ratings. The paper also estimates how many ratings a reliable comparison needs, and it releases the annotated data.

TL;DR

Professional translators re-annotated the WMT20 English-German and Chinese-English outputs with MQM. They marked each error span with a category and severity, and scored segments with fixed weights (Major 5, Minor 1) §3.3 tab. 1. The authors report that the resulting ranking differs from the WMT crowd ranking, that every human translation ranks above every MT system under MQM and pSQM, and that most metrics correlate better with MQM than with WMT §4.1 §4.6. All results are author-reported.

Problem

Human evaluation of high-quality MT is hard to do well. Segment ratings from crowd workers or researchers, each judged in isolation, can drown in rater noise once quality is high §1. The WMT20 campaign used researchers or translators for English-German and reference-based crowd workers for Chinese-English §3.1 §4.1. Earlier work had shown that professional translators can tell human from machine translations where crowd workers cannot §4.1, finding i §2. The paper argues that every score implicitly rests on identifying errors, so listing errors explicitly gives a base from which different scores can be derived for different uses §1.

Method

The paper compares three human evaluations on the same WMT20 outputs §3:

1. WMT baseline. Segment ratings on a 0-100 scale with document context. The organisers publish a raw score and a per-annotator z-score; the z-score is what the WMT column in the results reflects (inferred from the RAW column and the release README's "WMT DA" label, not stated in the paper) §3.1 README.md.
2. SQM. A 0-6 scale with defined levels at 0, 2, 4 and 6, and intermediate values allowed. It was run with crowd workers (cSQM) and with professional translators (pSQM) §3.2 §3.4.
3. MQM. Annotators highlight error spans, then assign each a category and a severity. The per-error labels become a weighted segment score §3.3.

MQM design §3.3 app. A app. B:

1. Error hierarchy. Five top-level categories: Accuracy, Fluency, Terminology, Style and Locale, with sub-categories app. B. A special Non-translation error was added after a pilot run §3.3.
2. Severity. Three levels: Major, Minor and Neutral. The paper drops the optional Critical level because its meaning is context-specific §3.3 app. B.
3. Annotation. Annotators label all errors in each segment, up to five target errors, and keep the five most severe if there are more. Source errors are annotated separately and do not count toward the five §3.3 app. B.
4. Non-translation. Used when a segment is too garbled to identify distinct errors. It covers the whole segment, and no other error is recorded for that segment §3.3 app. B.
5. Weights. Minor errors weigh 1 and Major errors weigh 5. The Major weight came from a resampling test of stability over 1 to 10 §3.3 app. B. Minor Fluency/Punctuation weighs 0.1, Neutral weighs 0, and Non-translation weighs 25 tab. 1.
6. Aggregation. A segment score is the weighted sum of its kept errors, averaged over annotators. Document and system scores are averages of segment scores §3.3 app. B.

The paper has no numbered equations. The scoring rule is given in prose and in Table 1 app. B, Scoring tab. 1.

# MQM segment score  [§3.3] [tab. 1] [app. B]
# Input:  errors_per_rater: one list of errors per annotator
# Output: segment score from 0 (perfect) to 25 (worst)

def mqm_segment_score(errors_per_rater):
    # Average the per-annotator scores  [§3.3] [app. B]
    rater_scores = []
    for errors in errors_per_rater:
        rater_scores.append(annotator_score(errors))
    return mean(rater_scores)


def annotator_score(errors):
    # Non-translation covers the whole segment and excludes other errors;
    # its weight is 25  [app. B] [tab. 1]
    if any(e.category == "Non-translation" for e in errors):
        return 25
    # Source errors are ignored and do not count toward the five-error
    # limit  [app. B]
    target = [e for e in errors if e.category != "Source error"]
    # Keep at most the five most severe target errors  [§3.3] [app. B]
    kept = most_severe(target, limit=5)
    # Sum the weights of the kept errors  [tab. 1]
    return sum(weight(e) for e in kept)


def weight(error):
    # Neutral errors weigh 0  [tab. 1]
    if error.severity == "Neutral":
        return 0
    # Minor Fluency/Punctuation weighs 0.1; other Minor errors weigh 1
    # [tab. 1] [app. B]
    if error.severity == "Minor":
        if error.category == "Fluency/Punctuation":
            return 0.1
        return 1
    # Major errors weigh 5, including Major Fluency/Punctuation  [tab. 1]
    return 5

Experimental setup §3.4 tab. 2:

• Data: the WMT20 English-German test set (1418 segments, 130 documents) and Chinese-English test set (2000 segments, 155 documents). Ten systems per pair, including the reference translations, drawn from the top WMT performers plus systems chosen to add diversity §3.4.
• Raters: professional translators, all native speakers of the target language. Pools for pSQM and MQM are disjoint §3.4.
• Assignment: documents are assigned round-robin to all 20 possible sets of three raters from the pool. Each rater annotates all ten systems for their documents, with anonymised systems and documents in random order §3.4.
• MQM rating count: three ratings per segment tab. 2. Segments per rater: 6,830 to 7,220 (English-German) and 9,860 to 10,210 (Chinese-English) §3.4.

Analysis steps §4: system scores (§4.1), error-category counts (§4.2), document-level scores (§4.3), rater agreement (§4.4), simulated rating budgets (§4.5), and re-scoring of WMT20 metrics with MQM (§4.6, app. C).

Hyperparameters

Learning rates, batch sizes and training schedules do not apply; the paper trains no model.

Results

Author-reported. Setup: WMT20 English-German and Chinese-English test sets, ten systems each, professional MQM with three raters per segment, baseline WMT20 crowd or researcher ratings §3 tab. 2.

Table 3, English-German tab. 3. Values only; the paper's ranks in parentheses are dropped. Lower is better for MQM and its breakdowns; higher is better for the others.

Table 4, Chinese-English tab. 4. Human-A has no WMT score in the source table (shown as "-").

The source table prints the Online-B WMT value as "0`.06"; it is read here as 0.06.

Error categories (condensed from tab. 5 English-German and tab. 6 Chinese-English). "Errors (%)" is the share of all error counts; "Major (%)" is the share of that category's errors that are Major. Human and All MT are MQM scores per segment; the ratio is All MT over human.

The English-German table has further rows (for example Fluency/Punctuation and Terminology/Inconsistent) that are omitted here; see tab. 5. Locale/Name format is listed only in the Chinese-English table tab. 6.

Rater agreement tab. 7 (MQM per rater, All row only; the ratio is each rater over the average):

Rating budgets tab. 8: MQM needs 951 ratings per system for English-German and 3720 for Chinese-English to reach an expected Kendall tau of 0.9 with the full data set §4.5. These are simulated values, from bootstrapping a multivariate Gaussian fitted to the covariance of all systems §4.5 fn. 2.

Metric correlations tab. 9. Averages over the WMT20 metrics. Sys-level: system-level; seg-level: segment-level; "+ human" rows add the human outputs to the systems being scored. p-values are omitted.

Headline findings, as the authors state them:

• Human translations rank above MT. Under pSQM and MQM, every human translation ranks above every MT system for both pairs §4.1, finding i tab. 3 tab. 4. Under MQM, the English-German MT score is about 2.9 times the human score (All categories) tab. 5. Human-P, a paraphrased reference, is 10th of 10 on WMT and 3rd on MQM for English-German tab. 3 §4.1, finding i.
• WMT ranking differs from MQM. Correlation between the WMT ranking and MQM is low for English-German and negative for Chinese-English at system level, and very low at segment level §4.1, finding ii fig. 1 fig. 2 fig. 3 fig. 4. The WMT Chinese-English evaluation was reference-based and monolingual, unlike the others §4.1, finding ii.
• pSQM is close to MQM at system level. The system rankings mostly agree, but segment-level Kendall correlation is about 0.5 §4.1, finding iii.
• MQM is driven by Major and Accuracy errors. Rankings from the Major-only and Accuracy-only sub-scores nearly match the full MQM ranking §4.1, finding iv tab. 3 tab. 4.
• Error patterns. The authors report that Accuracy/Mistranslation accounts for the majority of Major errors for both pairs §4.2 tab. 5 tab. 6. For English-German, sentences with more than five Major errors (Non-translation) are about 28 times as frequent for systems as for humans §4.2 tab. 5. The best systems are mostly penalised for mistranslation and Non-translation §4.2.
• Documents. Human translation quality is consistent across documents, while MT quality varies by document, and all MT systems share the same difficult documents §4.3 fig. 5 fig. 6.
• Metrics. Most metrics correlate better with MQM than with WMT, and almost all beat the WMT human correlation. The baseline metrics average is similar to the global average except for Chinese-English WMT §4.6 tab. 9. Adding human outputs to the systems lowers correlations sharply, especially under MQM §4.6 tab. 9. The paper names chrF at about 0.8 on English-German as an example §4.6.
• Rater agreement. The paper states that raters fall within ±20% of the mean for English-German and ±30% for Chinese-English, with rater 3 the most severe and rater 1 the most permissive there §4.4 tab. 7. The table conflicts with the ±20% figure; see Pitfalls.
• Rating budgets. For a fixed budget of 900 segment-level ratings, spreading ratings over more documents (three consecutive sentences per document) gives more accurate system scores §4.5 fig. 8.

Paper vs code

The repository linked from the paper (footnote 1, §1) is the authors' data release, not a scoring or analysis codebase. Its contents: TSV files of MQM and pSQM labels, per-segment average scores, and a README README.md mqm_newstest2020_ende.tsv. It has no scoring, weighting, correlation or simulation script. The README states the repository is "not an official Google product" README.md; it is the data release the paper names and points to, so we treat it as the paper's official repository.

• Sign of the average score. The paper's segment score runs from 0 (perfect) to 25 (worst) app. B, Scoring. The average-score file stores the negation. For segment 1 of system eTranslation.737, three rater rows carry Minor Style/Awkward (1), Major Accuracy/Mistranslation (5) and Minor Accuracy/Mistranslation (1) mqm_newstest2020_ende.tsv. Their sum is 7, which averages to 2.33 over three raters. The file stores -2.333333 mqm_newstest2020_ende.avg_seg_scores.tsv. Divide the average file by -1 before comparing with the paper.
• Category labels. The data uses "Non-translation!" with an exclamation mark mqm_newstest2020_ende.tsv. It uses "Locale convention/Currency format" mqm_newstest2020_ende.tsv, while Table 6 of the paper writes "Locale/Currency format" tab. 6. The paper's own hierarchy uses "Locale convention" app. B while its results tables use "Locale/..." tab. 6. Map labels by stem before counting.
• Scope. The README covers WMT20, WMT21 (newstest2021 and TED) and WMT22 and WMT23 (generalMT) releases README.md. The paper covers only WMT20.
• Numbers. The README's WMT20 tables give the same MQM means and WMT DA values as the paper's Tables 3 and 4, for example Tohoku-AIP-NTT 2.02 and DeepMind 5.41 README.md. The README labels the WMT column "WMT DA" README.md. We found no discrepancies in these values.
• Files not in the paper. The WMT20 folders hold "no-postedits" variants that the paper does not describe . We did not check what they remove.
• Rater bands. The text says all English-German raters fall within ±20% of the mean §4.4. The table gives Rater 3 at 1.23 of the average tab. 7, so the band is wider than the text says. See Pitfalls.

Pitfalls

• Scoring must be rebuilt. The paper states the weights and the five-error cap, but the repository holds no scoring script. Implementers must write the aggregation themselves. Ties in choosing the five most severe errors are not specified §3.3 app. B.
• Sign convention. Released average scores are negated relative to the paper's 0-to-25 scale mqm_newstest2020_ende.avg_seg_scores.tsv. Mixing the two conventions reverses every ranking.
• Major weight choice. The value 5 came from a stability analysis over 1 to 10 with resampling §3.3. The resampling procedure is summarised, not specified, and no code is released for it. Re-deriving the weight for a different data set needs a new analysis.
• Rater bands. The text says ±20% for English-German §4.4, but the All row shows Rater 3 at 1.23 tab. 7. Do not rely on the ±20% figure without checking the table.
• Category naming. The paper's section 4.2 says the taxonomy has four top-level categories, while section 3.3 and Appendix B list five (adding Locale) §4.2 §3.3. The released labels differ from the paper's labels (see Paper vs code).
• Segment-level thresholds differ. The WMT segment-level measure drops pairs whose raw scores differ by less than 25. For MQM, the authors use a threshold of 0 app. C. Segment-level correlations are therefore not directly comparable between WMT and MQM.
• Different WMT protocols. The Chinese-English WMT evaluation was reference-based and monolingual, while the English-German one used source-based ratings §3.1 §4.1. The paper's comparison does not separate the effect of protocol from the effect of rater quality.
• Metric analysis is small. Only seven English-German systems were rated for the metric analysis §4.6 fn. 3. System-level correlations on so few systems are noisy. The same footnote explains the use of Kendall tau over the official Pearson measure §4.6 fn. 3.
• Simulated budgets rest on a model. The 951 and 3720 rating counts come from a Gaussian bootstrap of the observed covariance, not from new annotation §4.5 fn. 2. Treat them as estimates.
• Cost figures are generic. Appendix A quotes an approximate MQM cost of about 1 USD per segment with three annotators. The paper presents this as a general estimate, not a measured cost of this study app. A.
• Post-editing check covers one pair. The paper checks that Chinese-English references were not post-edited MT, and says they came from translators with translation-memory tools §4.3. It reports no equivalent check for English-German.

Lineage

• Builds on: the Multidimensional Quality Metrics framework from the EU QTLaunchPad and QT21 projects §2 (no card in this wiki). The authors use the MQM usage guidelines for scientific research §3.3 app. A.
• Compared against: the WMT20 human evaluation §3.1, SQM with crowd and professional raters §3.2 §3.4, and the WMT20 metrics task §4.6.
• Superseded by: not known from content already in this wiki.