The Authenticity Gap in Human Evaluation

In plain words

The usual way to compare two text generators is to have people rate a sample of each generator's outputs on a 1-to-5 scale, average the ratings, and declare the higher average the winner. The authors ask whether that average really reflects what people prefer. Their answer is that it often does not: the scale may not mean the same thing to each person, and averaging can flip the answer when the two generators are close.

They propose a different question. Instead of asking for a score for each output, they ask each person how likely it is that they prefer system X to system Y after reading a set of outputs from both. Those probabilities are easy to combine across people. On story generation, the authors report that this recovers the ordering they expected from GPT-3 model sizes, while the standard rating-based method does not. The numbers come from the authors' own study.

TL;DR

• Standard human evaluation averages Likert ratings per system. The paper argues this only captures preferences under assumptions that are often violated, and that Likert averages can reverse the true preference direction §3.
• SPA asks each annotator for P[X ≻ Y] after exposure to a fixed sample of outputs from both systems, then averages those probabilities and tests them against 0.5 §5.2.
• Author-reported: on GPT-3 story generation with 90 annotators, SPA recovers 5 of 5 expected preferences; the standard protocol recovers 2 of 5 §6.1, tab. 1.

Problem

Human ratings are treated as the gold standard in NLG evaluation, and the standard protocol (outcome-level absolute assessment, OAA) collects ratings of sampled outputs, averages them per system and ranks systems by the average §1. The paper argues that this implicitly estimates an expected utility, which is valid only if annotators satisfy the axioms of von Neumann–Morgenstern (VNM) rationality: completeness, transitivity, continuity and independence [§2.4, Definition 6, Axioms 1–4]. For open-ended tasks such as story generation, the paper also says the additional condition needed to average across annotators (extremum equivalence) usually cannot be met §3, Remark 4.

Method

The paper has two parts: a theoretical critique of the standard protocol, and a new protocol with an experiment.

Framing (§2). Each system is a lottery over output text, given a prompt distribution, so a system is a compound lottery that reduces to a simple one by marginalising over prompts §2.1. Ratings are read as cardinal utility estimates under OAA. Averaging them is an estimate of expected utility, and that is justified only under VNM rationality §2.3, §2.4.

Causes of misrepresentation (§3). The paper lists four problems.

1. Ordinal–cardinal conflation (Remark 1). A Likert scale is ordinal, but averaging it treats the gaps between points as equal. The paper reports that in a survey, 31 of 38 NLG papers using Likert scales took an interval interpretation, and one justified it §3. Normalising with z-scores does not fix asymmetric intervals §3.
2. Biased estimation (Remark 2, Propositions 1–2). Under a best-case model in which the annotator treats Likert points as equal intervals and rounds up (ceiling), the rating equals the utility plus a residual r_i(s) = ⌈u_i(s)⌉ − u_i(s) §3. Propositions 1–2 state that a difference in mean residuals can make the averaged Likert rating over- or under-estimate one system, and that it can reverse the true preference when the expected-utility gap is smaller than the residual gap. Because r lies in [0, 1), the residual gap is at most 1, so reversals can only happen when the two systems are similar in average utility §3. The paper notes this situation is common for incremental improvements §3.
3. Non-independent lotteries (Remark 3). Independence (Axiom 4) fails when mixing a third lottery, such as offensive output, into both systems makes both unusable §3.
4. Inter-annotator incomparability (Remark 4). Using the same scale does not make annotators' magnitudes equal. Estimating the expected expected utility across annotators needs extremum equivalence (Boutilier et al., 2003): all annotators agree on their best and worst outcomes and assign them utilities c_max > c_min ≥ 0 §3. The paper argues this holds for machine translation with correct and incorrect references, but not for open-ended tasks §3.

The expected expected utility is written as

\text{EEU}[X] = \int \mathbb{E}[u_i(X')]\, p(u_i)\, du_i \qquad \text{[eq. 1]}

Standard-protocol fixes (§4). The paper suggests three changes that make OAA more sound for some tasks: continuous scales instead of Likert, maximum- and minimum-utility references, and no outcome that makes a system unusable. It notes that WMT has used a protocol with several of these features since 2017, but that it cannot be applied to open-ended tasks §4.

System-level probabilistic assessment (§5). SPA sidesteps aggregating outcome preferences into system preferences. Each annotator estimates the probability P[X ≻_i Y] that they prefer system X to system Y §5, intro.

• Theory (§5.1). The population probability is written as

  P[X \succ Y] = \int P[X \succ_i Y]\, p(\succ_i)\, d\succ_i \qquad \text{[eq. 2]}

  Because each value lies in [0, 1], the values can be averaged across annotators directly. The direction of the preference is set by whether P[X ≻ Y] is above or below 0.5, and the paper uses significance testing for the strength of evidence §5.1.

• Preferences are modelled as stochastic. An annotator's stated probability changes as they read more outputs, so each (annotator, exposure level t) pair is treated as a separate agent; at t = 0, P = 0.5 §5.1.

• SPA needs two assumptions: unbiased stated preferences (noise has expectation zero), and complementarity and indifference (P[X ≻ Y] + P[Y ≻ X] = 1, and indifference exactly when both are 0.5) §5.1, Assumptions 1–2. The paper says that the standard protocol needs VNM rationality, completeness, transitivity, continuity, independence and extremum equivalence, and SPA does not §5.1.

• Limits (§5.2 and Limitations section). SPA measures whether a preference exists, not its size. The paper notes that magnitude can still be estimated with a single annotator [Limitations, item 1]. Annotators may not understand probability, which the optional filter in step 3 addresses. Aggregating several quality axes is left to the annotator, who should judge overall preference directly [Limitations, item 3].

Implementation procedure

The paper's steps are §5.2. Step 1 draws the annotator pool and prompts. Step 2 elicits the probability. Step 3 is optional filtering. Step 4 aggregates and tests.

# Draw m prompts from the task's prior and sample one output per prompt
# from each system; give every annotator the same sample  [§5.2 step 1]
prompts = sample_prompts(task_prior, m)
for annotator in annotators:
    shown_x = [sample_output(system_x, p) for p in prompts]
    shown_y = [sample_output(system_y, p) for p in prompts]

# Ask for a probability on a 0-100 scale, not a choice  [§5.2 step 2]
for annotator in annotators:
    p_xy[annotator] = ask_chance_x_better(annotator, shown_x, shown_y)

# Optional: drop annotators whose two answers sum above tau = 1.1,
# and in multi-system tests drop them if they fail even once  [§5.2 step 3]
kept = [a for a in annotators if p_xy_rev[a] + p_xy[a] <= 1.1]

# Average the probabilities and test against chance (0.5)  [§5.2 step 4]
mean_p = average(p_xy[a] for a in kept)
p_value = two_sided_t_test(p_xy[kept], null_mean=0.5)

Hyperparameters

Results

Author-reported. Setup: GPT-3 story generation, 90 analysed annotators, five writers per prompt (four GPT-3 models and one human writer), each annotator reads five prompts §6.1, app. A. Table 1 shows the results. "Expected" follows the authors' prior: davinci ≻ curie ≻ babbage ≻ ada, and human ≻ ada and human ~ davinci §6.1.

Table 1 caption: SPA is significant in the expected direction for the four rows that expect a difference, and not significant for the indifference row (human versus davinci). The standard protocol is significant in the expected direction for two rows. Its Likert value for human versus davinci is significant in the unexpected direction (marked red in the paper). The paper's text says SPA recovers 5 of 5 expected preferences and the standard protocol 2 of 5 §6.1. The sign convention for Δ is not stated explicitly. The table has no effect sizes beyond P and Δ, and no confidence intervals.

DALL-E image generation (§6.2, fig. 1, app. B). The authors report that scrambling the prompt yields significantly worse images at α = 0.01 with Holm–Bonferroni correction, as expected fig. 1. Increasing the number of examples m makes annotators more certain, with diminishing returns between 4 and 8 examples. At m = 1 annotators are over-confident, which the authors attribute to not seeing variance within a single system §6.2. Swapping the question order gives probabilities that are not significantly different in distance from 0.5 §6.2. Exact probabilities are plotted in fig. 1 and not tabulated, so they are not stated in the text.

Filtering (app. B.2, fig. 5). The authors report that the τ = 1.1 filter increases effect sizes and makes P[X ≻ Y] and P[Y ≻ X] more symmetric [app. B.2, fig. 5]. Their recommendation is to use it app. B.2.

Statistical power (§6.3, fig. 2). At the observed effect sizes, the power of all tests is approximately 1 for α = 0.001 with n_A = 90, for both protocols §6.3, fig. 2. The authors take this to show that the standard protocol's failure is not a precision problem but a measurement problem §6.3.

Paper vs code

No official code. The paper has no code link. A GitHub search by method name, title words and author names found no repository from the authors.

Pitfalls

• The probability question is noisy. Asking for P[X ≻ Y] and then for P[Y ≻ X] does not give exactly complementary numbers §5.2 step 2. The τ = 1.1 filter is optional. The GPT-3 study did not apply it app. A.2, and the DALL-E study found it helpful app. B.2. An implementer should decide in advance whether to filter and report the choice.
• Exposure changes the answer. An annotator's probability depends on how many outputs they have seen §5.1. The authors used m = 5 for GPT-3 and recommend m ≥ 4 from their DALL-E study only §6.2. The optimal m for other tasks is not stated.
• Same prompts for both systems. Using different prompts for X and Y is allowed, but the authors say it makes the comparison harder and annotators less certain §5.2 step 1.
• Significance test details are underspecified. The paper says "two-sided Student's t-test" against 0.5 §5.2 step 4. Which annotators enter each test, and whether each comparison uses one probability per annotator, is inferred from the study design app. A.2. The aggregate uses one probability per annotator per comparison, yielding 90 values per test in the GPT-3 study.
• Multiple comparisons. Holm–Bonferroni correction is applied across the comparisons §6.1. The number of comparisons used for correction in the DALL-E study is not stated explicitly.
• Unique annotators. The method assumes n_A distinct annotators. The authors excluded annotators who submitted multiple HITs, because this controls how much each annotator sees app. A.2.
• Ambiguous attrition in the DALL-E study. "The filtering left 29 eligible annotators" app. B.2 does not say whether this figure already includes the other exclusions (non-native speakers and multiple submissions). Plan for heavy attrition: 60 recruited, 29 kept.
• Standard-protocol comparison is small. Each annotator rated only the first continuation from each writer, not all 25 app. A.2. This matches SPA's sample size but gives little within-annotator information for the standard protocol.
• Length confound control. The human story was trimmed to the longest machine story per prompt, to remove length as a cue app. A.1. Implementers must do this per prompt, not across the whole set.
• Expected preferences are priors. The expected orderings come from earlier work on GPT-3 and on Turing-style discrimination §6.1. The paper treats them as expectations, not ground truth.
• Model versions. The GPT-3 versions and DALL-E version are the ones named in the appendix. The DALL-E version is not stated §6.2.
• Scope. The paper focuses on NLG evaluation. It is written as an analysis of one protocol; the authors do not evaluate SPA on tasks beyond the two experiments.
• The authors' conclusions depend on their assumptions. The proofs rely on VNM rationality, continuity, and the given Likert model. The paper argues that SPA needs only the two assumptions in §5.1, but those assumptions are themselves unverified in the study.

Lineage

• Builds on: VNM expected utility (von Neumann and Morgenstern); Bradley–Terry and Thurstone comparison models §2.4; extremum equivalence (Boutilier et al., 2003) §3; Student's t-test and Holm–Bonferroni correction §6.1.
• Compared against: the standard OAA protocol with Likert ratings §1, §3; and outcome-level relative assessment (ORA), which the paper says is less common now §2.4, §3.
• Related: the WMT move to direct assessment §4, §7; GPT-3 model-size experiments §1, §6.1; the Turing-style discrimination studies of Clark et al. (2021) §1, §6.1.
• Superseded by: not stated.