All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text

In plain words

Researchers asked crowd workers to read short texts and say whether a person or a machine wrote each one. The machine texts came from GPT-3, a large language model, and from GPT-2, its predecessor. Stories, news articles and recipes were the three kinds of text used.

Without any training, the workers could not tell GPT-3 text from human text: their guesses were no better than a coin flip. Three quick training methods helped a little. Only the one that showed workers labelled example texts gave a clear overall gain, and most of that gain came from the story texts. The workers mostly judged spelling, grammar and style. Those are the features that current models already get right, so the authors argue that a quick, untrained human test measures the wrong thing for today's models. They suggest more careful evaluations that check content and usefulness.

TL;DR

Untrained crowd raters could not tell GPT-3 text from human text in stories, news or recipes (49.9% against 50% chance) §2.4 tab. 1. Three short training methods were tried; only labelled examples gave a significant overall gain (55%), and that gain was not significant within any single domain §3.2 tab. 2. The authors advise against small, untrained human evaluations of current models §5.

Problem

Human evaluation is the usual way to judge generated text, but most such evaluations ask for holistic quality ratings with little training §1. As models get more fluent, their errors are often about content, such as factual errors or inconsistency with the context, and a superficial read does not catch them §1. The paper asks two questions: can untrained evaluators tell GPT-2 and GPT-3 text from human text §2, and can cheap training improve that §3? The study uses non-expert raters on Amazon Mechanical Turk, the small-batch setting the authors say most NLG evaluations use §1 §2.3.

Method

This is an empirical study, not a new model. It has three parts: building the human and machine texts (§2.2), running the rating task (§2.1, §2.3), and running the training conditions (§3.1). Results are analysed in §2.4 and §3.2, and free-text reasons are coded in §2.5 and §3.3.

1. Human texts §2.2. For each of three domains, 50 human-written texts were used. Stories came from Reddit WritingPrompts: 255 stories beginning with "Once upon a time", of which 50 were sampled §2.2.1. News came from 2,111 local articles from 15 newspapers, filtered to remove articles under 100 words, non-local articles and coronavirus articles; 50 were used as human texts and 50 as prompts §2.2.2 app. A.1. Recipes came from RecipeNLG; 50 were used as human texts and 50 as prompts §2.2.3.
2. Machine texts §2.2. Each domain has the same three priming texts, human-written and in-domain. Each generated text is conditioned on them in the "three-shot" setting from the GPT-3 paper §2.2. The domain prompt is "Once upon a time" for stories, the headline and first sentence of a prompt article for news, and the title and ingredient list for recipes §2.2.1 §2.2.2 §2.2.3. Sampling uses temperature 0.7 §2.2.
3. Length and cut-off §2.2. Only texts of 100 words or more are kept. After 100 words, a text is cut at the end of the next sentence (sentence splitting with NLTK).
4. Copy check §2.2.1. A story that copies a priming text with more than 80% overlap is regenerated (9 GPT-2 and 2 GPT-3 stories were regenerated).
5. Rating task §2.1 §2.3. A rater is assigned one domain and one model. Each rater reads 5 texts, a mix of human and model texts, and picks one of one of four graded labels: two for human (confident or hedged) and two for machine (hedged or confident). After a confident-human answer the rater is asked why; otherwise they are asked what they would change to make the text seem more human-like §2.1 fig. 2.
6. Analysis §2.4. Responses are binned into human and machine. Accuracy is the share of correct source guesses. The paper also reports F1, precision, recall, Krippendorff's alpha, the share of "human" guesses and the share of "definitely" answers tab. 1.

# Human-vs-machine rating study  [§2.1-§2.4]
# Input:  human_texts, prompts, priming_texts (3 per domain)
# Output: per-rating accuracy by domain and model

def make_machine_text(domain, prompt, priming_texts, model):
    # Three in-domain human texts come first  [§2.2]
    context = join_with_eos(priming_texts)
    # Domain prompt, appended after the priming texts  [§2.2.1-§2.2.3]
    context = context + prompt
    # Sample at temperature 0.7  [§2.2]
    text = sample(model, context, temperature=0.7)
    # Stories that copy a priming text are regenerated  [§2.2.1]
    while domain == "stories" and overlap(text, priming_texts) > 0.8:
        text = sample(model, context, temperature=0.7)
    # Keep texts of 100+ words, cut after the next sentence end  [§2.2]
    return cut_after_next_sentence(text, min_words=100)

def accuracy(ratings, true_sources):
    # Collapse the 4-point scale: labels 1-2 mean human, 3-4 mean machine
    # (the scale order implies this; the paper states only the binning)
    # [§2.1] [§2.4 footnote]
    guesses = ["human" if r <= 2 else "machine" for r in ratings]
    # Share of guesses that match the true source  [§2.4]
    return mean(g == s for g, s in zip(guesses, true_sources))

Key equations: none are numbered. The significance test is a two-sided t-test with a Bonferroni-corrected threshold of p < 0.00333 tab. 1. The number of comparisons behind that threshold is not stated.

Training conditions (§3): the study is re-run with GPT-3 texts only and the same texts as §2 §3.1, with one of three trainings in front of the task §3.1:

• Instructions: the task instructions also name repetition and factuality as useful cues and grammar, spelling and style as misleading cues §3.1.1. Cost: US$1.25 per task §3.1.1.
• Examples: three practice rounds. Each round shows one text with its true source and a short explanation after the rater guesses §3.1.2. Cost: US$1.75 per task §3.1.2.
• Comparison: the Examples texts, each paired with a text of the other source written from the same prompt. The rater picks the machine text, then sees the answer and explanation §3.1.3. Cost: US$2.25 per task §3.1.3.

Annotation (§2.5, §3.3, app. A.3): 150 free-text reasons per condition were coded by two authors each. Labels fall into form (grammar, level of detail, genre), content (repetition, factuality, consistency, common sense, coherence) and machine capabilities tab. 4. Several labels per comment are allowed.

Hyperparameters

Results

Author-reported. Setup: MTurk raters, three domains, two generators (GPT-2-XL and GPT-3 Davinci), 5 texts per rater, 780 raters in §2 and 1,170 in §3 §2.3 §3.1. The baseline is chance (50%) for detection §2.4.

Headline numbers:

• GPT-2 text: 57.9% correct overall, significantly above chance (t = 6.58, p < 0.0001) §2.4 tab. 1.
• GPT-3 text: 49.9% correct overall, not different from chance (t = -0.09, p = 0.93) §2.4 tab. 1.
• GPT-3 accuracy does not differ by domain (ANOVA F = 0.78, p = 0.46) §2.4.
• Story accuracy fell from 62% (GPT-2) to 48% (GPT-3), Cohen's d = 0.57 §2.4.
• Examples training: 55% overall, the only significant gain over no training (Tukey-adjusted p < 0.003, d = 0.25) §3.2. Within domains, no gain is significant §3.2. Instructions (52%) and Comparison (53%) are not significant tab. 2.
• Krippendorff's alpha is at most 0.11 in every training setting §3.2 tab. 2.

Table 1. Untrained raters, §2 (author-reported, from the paper's tab. 1). An asterisk marks accuracy significantly better than chance.

Table 2. Training conditions, GPT-3 texts only, §3 (author-reported, from the paper's tab. 2). "None" is the GPT-3 rows of Table 1. An asterisk marks accuracy significantly better than None.

Reason coding (author-reported, from the paper's tab. 3; share of label mentions, not of raters):

Examples and Comparison training both gave the highest recall and F1 and lowered the share of "human" guesses in all three domains, so raters were willing to call more texts machine-written §3.2. The authors' reading is that this is a change of focus, not proof of correct judgment §3.3.

Paper vs code

No official code. The paper has no model or analysis code. Its study materials (the priming texts and the training examples) are said to be in the supplementary material and on a project page §2.2 §3.1. The project page could not be read during compilation. A GitHub search found eaclark07/nlg_human_evals, described as supplementary materials for this paper; its owner handle is close to an author's email handle, which is not enough to verify it as official. It has no LICENSE file, so its contents were not used, and it is not recorded as code. Nothing was compared against it.

Pitfalls

• Decoding for GPT-2 is not stated. The paper gives temperature 0.7 under "default GPT3 generation settings" §2.2. It is unclear whether GPT-2-XL was sampled the same way. The paper also calls the three-shot setting "not typically how GPT2 is used in practice" §2.2.
• Length rule scope. "We only considered texts 100 words or longer" is written in the data section without saying whether it applies to human texts too §2.2.
• Copy-overlap measure undefined. The 80% overlap test for regenerating stories has no stated metric §2.2.1.
• Binning of the 4-point scale. The paper says responses were binned into two categories but not which labels map where. The natural split, labels 1-2 as human and 3-4 as machine, is inferred from the scale order §2.1 §2.4 footnote.
• Rounding. The text gives 49.9% for GPT-3, and the table gives 0.50. Similarly, the text gives 0.019 and 0.062 for the news and recipe gains §3.2, while the rounded table values give 0.02 and 0.06. Recompute from raw data, not from the table.
• Comparison baseline. Training conditions are compared with the "None" rows from §2, which come from a separate set of raters §3.1 tab. 2. The paper uses them as the baseline. Because the raters differ, the comparison is between groups, not within raters.
• Agreement is very low. Krippendorff's alpha is near zero in every setting §2.4 §3.2. A single rater's guess is noisy, so per-rating accuracy is a poor signal for one person.
• Significance is domain-sensitive. The Examples gain is significant overall but not within any domain. Most of it comes from stories (0.086 gain against 0.019 for news and 0.062 for recipes) §3.2.
• Filtering uses judgement. Raters were rejected for copied, mismatched, vague or uninterpretable explanations §2.3. The criteria are partly subjective, and 18.6% of raters were rejected §2.3.
• Interactive variant failed. A fourth condition, where raters rewrote the text to sound human, had a retention rate under 30% and a rejection rate over 50% on Amazon Mechanical Turk at this price §5 footnote.
• Results are from 2021 model access. GPT-3 Davinci was accessed through an academic API programme [Acknowledgments]. The paper does not say which API snapshot was used. Current model results cannot be read off this paper.
• Pilot for training choice. The pilot had 22 MTurk raters and 11 expert raters (NLP graduate students). Its findings shaped the instruction dimensions app. A.5 §3.1.1.

Lineage

• Builds on: GPT-3 (Brown et al., NeurIPS 2020) for the three-shot generation setting and the Davinci model §2.2; GPT-2-XL for the second generator §2.2; the four-point rating scale and the instruction dimensions of Ippolito et al. (2020, "Automatic Detection of Generated Text is Easiest when Humans are Fooled") §2.1 §3.1.1.
• Compared against (not directly comparable, per the paper): Ippolito et al. (2020), with trained raters at 71.4% on open-ended GPT-2-L text; Garbacea et al. (2019, "Judge the Judges"), with 66.6% on product reviews; and the GPT-3 paper's news-source task, at 52% §6.
• Superseded by: not stated in this paper.