--- kind: concept slug: human-evaluation title: Human evaluation summary: >- Judging generated outputs, dialogues or robot behaviour with people and turning the judgments into a verdict. Averaged Likert ratings can reverse the true preference when systems are close; a per-annotator probability question recovered more expected orderings in one study. Pairwise trials feed a ranking model. Results are author-reported, from twenty-one papers. tags: [evaluation, human-evaluation, nlg-evaluation] papers: ["1808.09637", "1812.01717", "2009.14306", "2104.13346", "2104.14478", "2107.00061", "2205.11930", "2212.14882", "2301.13848", "2303.11897", "2305.01569", "2305.09617", "2307.03109", "2310.13548", "2402.17937", "2405.17739", "2410.18727", "2412.05024", "2501.09024", "2505.17613", "2506.18123"] related: [llm-as-a-judge, llm-evaluation, benchmark-validity, dialogue-evaluation, dialogue-generation, bradley-terry-model, rlhf, mixed-effects-model, text-to-image-generation, text-summarization, generative-model-evaluation, neural-machine-translation, ai-generated-text-detection, psychometric-scale-validation, teleoperation, hallucination-detection, text-summarization] status: draft updated_at: 2026-11-01 --- ## Overview Human ratings are often treated as the reference for judging generated text, dialogue or robot behaviour. The question is how to turn many individual judgments into a verdict on which system is better, or into a finding about why something happened. Twenty-one cards on this wiki use human judgments in different ways. One argues that the usual average of Likert ratings can reverse the true preference [arxiv:2205.11930 §3]. Others rate single outputs for human-likeness [arxiv:1808.09637 §4.4], ask for pairwise preferences [arxiv:2009.14306 §6.3], rank policies from double-blind pairwise trials [arxiv:2506.18123 §3.1], code what people said during a session [arxiv:2405.17739 §3.4], collect pairwise image picks from users at scale [arxiv:2305.01569 §2], have crowd workers choose between a truthful and a sycophantic reply [arxiv:2310.13548 §4.3], or follow users' outcomes over 15 days with questionnaires [arxiv:2402.17937 §2.5]. Three robot cards add three more designs: a task-outcome study with workload and trust questionnaires across three arms [arxiv:2410.18727 §IV], a within-subject rating of a robot's regional and standard speech [arxiv:2412.05024 §5.1], and a 1-to-5 rating of a vision-language model's answers by fifteen judges [arxiv:2501.09024 §IV-B]. A summarisation card has annotators rate LLM summaries on several aspects, then run blinded pairwise tests [arxiv:2301.13848 §3.1] [§4.2], and a medical QA card has physicians rank long answers pairwise on nine axes [arxiv:2305.09617 §3.5]. A survey card lists the human criteria used for LLMs and says when a criterion needs people at all [arxiv:2307.03109 §5]. Three more cards differ in purpose. A radiology study has 15 radiologists rate ChatGPT rewrites of medical reports for errors and harm [arxiv:2212.14882 §3.3], and a text-to-image metric is checked against human ratings of how well each image matches its prompt [arxiv:2303.11897 §5.1], and a video-metric card runs pairwise human comparisons to check a set-level metric against frame-level ones [arxiv:1812.01717 §4.3]. Three cards cover error labelling and detection. In translation, professional raters mark each error span with a category and a severity, and the marks become a weighted segment score [arxiv:2104.14478 §3.3]. In text generation, untrained crowd workers say whether a short text was written by a person or by GPT-3 or GPT-2, and a few training methods are tested against that baseline [arxiv:2107.00061 §2.1] [§3.1]. In summarisation, crowd workers label each sentence of a summary as factual or not, and tag any error type [arxiv:2104.13346 §3]. Most variants share a skeleton: a fixed sample of outputs from each system, judged by several annotators, and a statistical test or model at the end. They differ in what the annotator is asked and in how the answers are combined. For an LLM used as the annotator, see [LLM-as-a-judge](/concepts/llm-as-a-judge). All numbers on this page are the authors' own and come from studies on specific tasks, several of them small. No card tests one variant against another on the same task, except where a card reports its own baseline. ## Variants ### Absolute ratings averaged per system (standard Likert protocol) Each annotator rates single outputs on a Likert scale. The ratings are averaged per system and the systems are ranked by that average [arxiv:2205.11930 §1]. The authors say the average estimates expected utility only if annotators satisfy the von Neumann–Morgenstern axioms and agree on their best and worst outcomes, so that ratings can be pooled [arxiv:2205.11930 §2.4] [§3]. On GPT-3 story generation, this protocol recovered 2 of 5 expected preferences [arxiv:2205.11930 tab. 1]. Card: [The Authenticity Gap in Human Evaluation](/papers/2205.11930). The same kind of rating appears in dialogue work. Crowd workers chatted with a bot or a human and rated the partner's human-likeness on a 1-to-5 scale. Human partners scored 4.3 on the new Craigslist data [arxiv:1808.09637 §4.4] [tab. 7]. Card: [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637). A five-point rating of fluency, consistency, naturalness, persuasiveness and engagingness was also collected with 25 participants per model [arxiv:2009.14306 app. E] [tab. 10]. Card: [INSPIRED: Toward Sociable Recommendation Dialog Systems](/papers/2009.14306). A multi-aspect version scores each summary for faithfulness as a yes-or-no judgment, and for coherence and relevance on a 1-to-5 scale. Three annotators rate each summary and the ratings are averaged [arxiv:2301.13848 §3.1]. Ten LLMs were scored this way on two news datasets, next to the existing reference summaries and two fine-tuned systems [arxiv:2301.13848 tab. 2]. Card: [Benchmarking Large Language Models for News Summarization](/papers/2301.13848). A small robot-language study uses the same 1-to-5 scoring with a panel of fifteen judges. Each answer to 50 scene questions is scored, and the scores are averaged per task type for three models. The two baselines are prompted only, and the proposed model is fine-tuned [arxiv:2501.09024 §IV-B] [tab. 1]. The authors call the gaps significant, but the card finds no test, variance or interval in the paper [arxiv:2501.09024 §IV-B]. Card: [Social-LLaVA](/papers/2501.09024). ### Repaired absolute protocol The same per-output ratings, with three changes: a continuous scale instead of Likert points, maximum and minimum reference outcomes shown to annotators, and no outcome that makes a system unusable [arxiv:2205.11930 §4]. The paper notes that a protocol with several of these features has been used for machine translation since 2017, and that such a protocol cannot be applied to open-ended tasks [arxiv:2205.11930 §4]. No experiment in the card tests this variant. Card: [The Authenticity Gap in Human Evaluation](/papers/2205.11930). ### System-level probabilistic assessment (SPA) Each annotator reads a fixed sample of outputs from both systems and states the percent chance, from 0 to 100, that system X is the better writer. The probabilities are averaged across annotators and tested against 0.5 [arxiv:2205.11930 §5.2]. Because each answer lies between 0 and 1, the answers can be averaged directly, with no pooling of scale values. The authors say SPA needs two assumptions: stated preferences are unbiased on average, and preferences are complementary (P[X ≻ Y] + P[Y ≻ X] = 1), with indifference at 0.5 [arxiv:2205.11930 §5.1]. The standard protocol needs the stronger conditions listed above; SPA does not [arxiv:2205.11930 §5.1]. Evidence from the authors' GPT-3 study, with the expected ordering from their prior [arxiv:2205.11930 §6.1]: | System X | System Y | Expected | SPA P[X ≻ Y] | Likert Δ (standard) | |---|---|---|---|---| | GPT-3-ada | human | X ≺ Y | 0.420* | −0.822*** | | GPT-3-babbage | GPT-3-ada | X ≻ Y | 0.688*** | 0.644*** | | GPT-3-curie | GPT-3-babbage | X ≻ Y | 0.630*** | 0.322 | | GPT-3-davinci | GPT-3-curie | X ≻ Y | 0.575*** | 0.244 | | human | GPT-3-davinci | X ~ Y | 0.544 | −0.389* | Source: [arxiv:2205.11930 tab. 1]. SPA matched the expected direction in all five rows, and the standard protocol in two. The authors read the power of the tests as near 1 for both protocols at the observed effects, so they take the standard protocol's failure to be a measurement problem, not a lack of precision [arxiv:2205.11930 §6.3]. In a DALL-E image study, scrambling the prompt gave significantly worse images at α = 0.01 after correction, as expected [arxiv:2205.11930 §6.2]. Card: [The Authenticity Gap in Human Evaluation](/papers/2205.11930). ### Pairwise preference with a "can't tell" option Each annotator chats with or reads two systems and picks the better one on a named aspect, with an option to say they cannot tell. The INSPIRED study used this on five aspects with 28 participants, each chatting with both models two or three times [arxiv:2009.14306 §6.3]. The text reports that the strategy model was preferred on all five aspects, with the largest gap on engagingness, but gives no counts or test [arxiv:2009.14306 §6.3] [fig. 5]. Card: [INSPIRED](/papers/2009.14306). The summarisation study used a blinded pairwise test with a tie option, for overall preference and for informativeness. Each annotator judged the same 100 pairs [arxiv:2301.13848 §4.2]. Overall preference came out equal in aggregate between a zero-shot instruction-tuned model and freelance writers. The more abstractive summary was judged more informative 51.1% of the time, which the authors do not treat as significant [arxiv:2301.13848 §4.2]. Card: [Benchmarking Large Language Models for News Summarization](/papers/2301.13848). ### Multi-axis pairwise ranking by domain experts Physicians compare two answers to the same question and pick the better one, or a tie, on each of nine axes. The axes cover medical consensus, reading comprehension, knowledge recall, reasoning, unwanted and omitted content, demographic bias, and two harm measures [arxiv:2305.09617 §3.5]. Med-PaLM 2 answers were preferred to physician-written answers on eight of the nine axes, at p < 0.001. The exception was inaccurate or irrelevant content [arxiv:2305.09617 §4.2] [fig. 5]. The paper uses bootstrap intervals and two-tailed permutation tests, each with 10,000 iterations, and blocks the permutation test by answer when an answer has several ratings [arxiv:2305.09617 §3.5]. The number of raters per pair in this pairwise task is not stated [arxiv:2305.09617 §3.5]. Card: [Towards Expert-Level Medical Question Answering with Large Language Models](/papers/2305.09617). ### Double-blind pairwise trials with a ranking model Evaluators run head-to-head trials of two policies on tasks they choose. They do not know which policy is which. Each trial gives a progress score, a binary preference and a free-text reason [arxiv:2506.18123 §3.1]. The records are fit with a Bradley-Terry model with latent task buckets, by EM, with a Davidson tie term [arxiv:2506.18123 §3.2] [alg. 2] [app. B.1]. The authors report that this ranking is closer to an exhaustive oracle than the standard DROID protocol, and that it is stable after about 100 comparisons [arxiv:2506.18123 §5.2] [§5.3]. Card: [RoboArena](/papers/2506.18123). For the ranking model alone, see [Bradley-Terry model](/concepts/bradley-terry-model). ### Qualitative coding of observed sessions Researchers watch a person work, take notes line by line, and tag each line with a codebook. Two coders first code the same sessions, then discuss disagreements [arxiv:2405.17739 §3.4]. This answers why people struggle, not which system is better. The study of 21 novice programmers reached Cohen's kappa 0.74 on the first sessions and reported three new difficulty codes [arxiv:2405.17739 §3.4] [tab. 2]. Card: [The Widening Gap](/papers/2405.17739). ### Pairwise picks from open-platform users, with ties Users type a prompt, see two generated images, and pick one or call a tie. The rejected image is replaced and the loop repeats for that prompt [arxiv:2305.01569 §2]. The authors chose the two-image, tie-allowed design over four images without ties and over two without ties, after a pilot on engagement and inter-rater agreement [arxiv:2305.01569 §2]. The released data has 583,747 training examples [arxiv:2305.01569 §2]. A scorer trained on the picks predicted held-out user choices at 70.5%, against 68.0% for expert annotators [arxiv:2305.01569 tab. 1]. The authors then use this scorer to rank images from other models [arxiv:2305.01569 §5]. Card: [Pick-a-Pic](/papers/2305.01569). ### Forced choice by crowd workers on truthful versus sycophantic replies For each pair of replies, five crowd workers pick the one they prefer, without internet access [arxiv:2310.13548 §4.3] [app. D.3]. The study has 1,330 comparisons over 266 misconceptions [arxiv:2310.13548 app. D.3] [§4.3]. The text says truthful replies were usually preferred, less reliably at higher difficulty, and gives no number [arxiv:2310.13548 §4.3.1] [fig. 7b]. Card: [Towards Understanding Sycophancy](/papers/2310.13548). ### Field study with self-report instruments and a mixed model Participants use one condition for many days. Each session has a before and an after anxiety survey, and the study tracks a distress instrument on days 1, 8 and 15 [arxiv:2402.17937 §2.5]. The analysis uses a mixed-effects model with pre-session anxiety as a covariate [arxiv:2402.17937 §2.6.3] [§3.2.2]. The study had 38 completers across a robot, a chatbot and a worksheet group, over 15 days [arxiv:2402.17937 §2.3.3]. Distress fell in the robot and worksheet groups and not in the chatbot group [arxiv:2402.17937 §3.2.1]. Post-session anxiety was about 6 points lower in the robot and chatbot groups than in the worksheet group [arxiv:2402.17937 tab. 4]. Card: [LLM-Powered Socially Assistive Robot-Delivered CBT Exercises](/papers/2402.17937). ### Within-subject questionnaire comparison with t-tests The same participants experience both versions of one system, and each rates the result on a validated scale after every run. The Pepper study had 17 adults hear the same dialogue in High German and in Low German. After each dialogue they rated the 18 items of the Robotic Social Attributes Scale on a 5-point scale [arxiv:2412.05024 §5.1]. For each of three factors, a t-test compared the two versions at significance level 0.05 [arxiv:2412.05024 §5.2]. Warmth was higher for the Low German version (p = 0.027). Competence (p = 0.378) and discomfort (p = 0.056) were not significant [arxiv:2412.05024 §5.2]. The paper does not say whether the test was paired, although the design is within-subject [arxiv:2412.05024 §5.1] [§5.2]. Card: [Talking Like One of Us](/papers/2412.05024). ### Task outcomes and questionnaires across robots and groups Non-experts steer one robot arm through a digital copy of the robot, for ten minutes, to stack three cubes into a tower. Afterwards they answer a single-ease question, the raw NASA-TLX for workload, and five items from the NARS S1 subscale, which the authors use as a proxy for trust [arxiv:2410.18727 §IV] [§II-C]. The study had 74 people from the UK and Japan and three arms. The UR5e built a mean of 10.50 towers per user, against 3.25 for Baxter and 1.03 for the UR3, and it had the lowest mental workload [arxiv:2410.18727 §V-A] [§V-B]. The arm, its gripper, its controller and the country all differed together, so the comparison cannot separate the arm from the country [arxiv:2410.18727 §IV-A] [§VI]. Card: [Breaking Down the Barriers](/papers/2410.18727). ### Expert Likert ratings with highlighted errors, one generator Radiologists read three original reports and one randomly chosen simplified version of each. They rate three statements on a 5-point scale: the simplified report is factually correct, it includes the relevant information, and it could lead a patient to wrong conclusions and harm [arxiv:2212.14882 §3.3]. Each statement has a follow-up where the rater highlights wrong text, lists missing information or lists harmful conclusions [§3.3]. The authors report medians, quartiles and ranges per statement, then group the free-text answers by content [§3.4] [§4.2]. The study had 15 raters and 45 simplified reports from one generator and no baseline, so it describes one system rather than ranking several [§3.2] [§3.3]. Medians were 2 (agree) for correctness and completeness and 4 (disagree) for harm, and 51% of raters highlighted at least one incorrect passage [tab. 1] [tab. 3]. Card: [ChatGPT Makes Medicine Easy to Swallow](/papers/2212.14882). ### Human ratings as the reference for an automatic metric Here the human judgments are the ground truth for checking a metric, not the final result. Annotators rate each generated image from 1 to 5 for how well it matches its prompt. The TIFA study used 20 graduate students, 800 images from five generators on 160 prompts, and two annotators per image, with Krippendorff's alpha 0.67 [arxiv:2303.11897 §5.1] [app. C] [tab. 2]. The metric's Spearman correlation with these ratings was 59.7 with mPLUG as the VQA model, against 33.2 for CLIPScore [arxiv:2303.11897 tab. 2]. A second protocol has annotators answer the metric's own multiple-choice questions on the generated images. It adds a "None of the above" option, uses two annotators per question with a third on disagreement, and reports alpha 0.88 [arxiv:2303.11897 §5.4] [app. C]. Card: [TIFA](/papers/2303.11897). A video metric gets the same treatment in a different form. Raters compare two generated videos and pick the better one or call them similar quality. Model pairs are chosen so that they tie on one metric, or spread across its range, and each row of the agreement table holds one metric fixed [arxiv:1812.01717 §4.3] [tab. 1]. With SSIM held equal, FVD agreed with raters on 74.9% of comparisons, and with PSNR held equal on 81.0% [arxiv:1812.01717 tab. 1]. The human study used BAIR models from four families, with 10 models per condition [arxiv:1812.01717 §4.3]. Card: [Towards Accurate Generative Models of Video](/papers/1812.01717). Factuality metrics are checked against sentence-level labels in the same way. FRANK correlates each metric with the summary-level human score, after removing the effect of the system that wrote the summary [arxiv:2104.13346 §5.2]. On all data, the best partial Pearson correlation is 0.27, for BERTScore P, and the best Spearman is 0.30, for FactCC [arxiv:2104.13346 tab. 2]. Entailment-based metrics do best on CNN/DM, and their correlation on XSum is about a quarter of that [arxiv:2104.13346 §5.3] [fig. 4]. Card: [FRANK](/papers/2104.13346). A benchmark for generated images, audio and mixed outputs uses people to check its automatic judges, not to rank systems. Annotators label sampled outputs, and their labels are compared with the judges' decisions. Two annotators label each question and a third settles disagreements. Agreement is reported by modality, from 91.0% for interleaved audio-text to 96.6% for interleaved image-text, and 94.4% overall. The pool was 20 graduate students [arxiv:2505.17613 §4.1] [tab. 8]. The human check covers 41 tasks and 978 instructions, so the other tasks are not covered by it [arxiv:2505.17613 §1]. Card: [MMMG](/papers/2505.17613). ### Error-span annotation with severity weights (MQM) Professional annotators read a whole document, highlight each error span, and give each error a category and a severity. The paper's scheme has three severities (Major, Minor, Neutral) and error categories grouped under accuracy, fluency, terminology, style and locale [arxiv:2104.14478 §3.3] [app. B]. The segment score is a weighted sum. Major errors weigh 5, Minor 1, Neutral 0, and Non-translation 25 for a segment too garbled to annotate further. At most the five most severe target errors per annotator count [arxiv:2104.14478 tab. 1] [app. B]. Three professional raters annotate each segment [arxiv:2104.14478 tab. 2]. The authors report that every human translation ranks above every machine system under MQM and pSQM, in both language pairs [arxiv:2104.14478 §4.1]. The English-German data set has 1,418 segments and the Chinese-English one 2,000 [arxiv:2104.14478 §3.4]. Card: [Experts, Errors, and Context](/papers/2104.14478). ### Sentence-level factuality labels with an error typology, crowd majority Crowd workers read each sentence of a generated summary in the context of the full summary, and say whether it is factual. If not, they pick one or more error types from a seven-type typology, plus "Other" and "Not an error" [arxiv:2104.13346 §2] [§3]. Three annotators label each item. A sentence counts as wrong when a majority marks it, and a summary counts as wrong when any sentence does [arxiv:2104.13346 §4]. The authors ran a pilot that rated whole summaries on a Likert scale and dropped it, because they found it less precise and less consistent [arxiv:2104.13346 §3, footnote]. Workers needed at least 85% on a qualification test. During the task, workers were checked with entity questions and with altered sentences, and those below 50% on the altered sentences were blocked [arxiv:2104.13346 §3] [app. A.6]. The labelled set has about 2250 summaries from nine systems on two news datasets. The authors report that 92% of XSum summaries and 43% of CNN/DM summaries have at least one error [arxiv:2104.13346 §4]. The overall 60% figure could not be reproduced from the per-system rows, so treat it as unverified [arxiv:2104.13346 tab. 3]. Fleiss kappa is 0.58 for factual versus not, and 0.39 for the category choice when all three annotators agree a sentence is not factual [arxiv:2104.13346 §3]. Card: [FRANK](/papers/2104.13346). ### Untrained crowd detection of machine text, with optional training Crowd workers on Amazon Mechanical Turk read five texts from one domain and one model, a mix of human and machine text, and label each text on a four-point scale [arxiv:2107.00061 §2.1] [§2.3]. The authors compare this untrained task with three short trainings: instructions, worked examples, and a comparison of paired texts [arxiv:2107.00061 §3.1]. Only the worked examples gave a significant overall gain, to 55% against 50% [arxiv:2107.00061 §3.2]. Untrained raters were at chance on GPT-3 text (49.9%) and above chance on GPT-2 text (57.9%) [arxiv:2107.00061 §2.4] [tab. 1]. Card: [All That's "Human" Is Not Gold](/papers/2107.00061). ## When to use which | Situation | Use | Why | |---|---|---| | Open-ended output with no clear best and worst reference, such as stories | SPA | The authors argue the absolute protocol's conditions fail here, and SPA needs neither the rationality axioms nor the reference condition [arxiv:2205.11930 §3] [§5.1]. On GPT-3 stories, SPA matched 5 of 5 expected orderings [arxiv:2205.11930 tab. 1]. | | Two systems are close in average quality, such as an incremental improvement | SPA | The authors show a Likert average can flip the order when the true gap is smaller than the rounding residual, and they say such close pairs are common for incremental work [arxiv:2205.11930 §3]. | | Clear best and worst references exist and a continuous scale is possible, such as the paper's machine-translation case | Repaired absolute protocol | The authors argue that the reference condition holds for machine translation with correct and incorrect references, and that the fixes apply [arxiv:2205.11930 §3] [§4]. | | Long generated answers judged by experts on several named axes, with a tie option | Multi-axis pairwise ranking | Med-PaLM 2 was compared this way with physician answers, using a panel of 15 physician raters [arxiv:2305.09617 §3.5] [§4.2]. The card gives no rater count per pair, so set one before the run. | | Summaries from several LLMs, scored on aspects, against a reference set that may be weak | Per-aspect ratings, plus a pairwise test against a human baseline | The summarisation study found the existing references rated below most systems, so reference-based comparisons are unreliable [arxiv:2301.13848 §3.2] [§3.3]. Its pairwise agreement was low, so report it with the result [arxiv:2301.13848 §4.2]. | | Interactive system, many variants, you want an ordering of all of them | Pairwise trials with a ranking model | RoboArena's double-blind design does not need the same tasks across trials and fits all records into one ranking; the authors report stable rankings after about 100 comparisons [arxiv:2506.18123 §5.3]. Their setting is real robots, so the transfer to text is not tested. | | Aspects you want to compare in a dialogue, with a small pool | Pairwise with "can't tell" | Gives a per-aspect direction without a rating scale [arxiv:2009.14306 §6.3]. The study has no reported test, so treat the direction as a hint. | | You need the size of the gap, not only its direction | SPA, with care | SPA measures whether a preference exists, not its size. The paper says magnitude can still be estimated with a single annotator [arxiv:2205.11930 limitations]. | | Quality has several axes you want scored separately | Decide before running | SPA asks for overall preference, and the paper leaves combining the axes to the annotator [arxiv:2205.11930 limitations]. The card gives no per-axis method. | | You need to know why people succeed or fail | Qualitative coding | Coding the notes gives categories, and the authors check coder agreement [arxiv:2405.17739 §3.4]. It does not rank systems. | | A criterion you are unsure needs people at all | Automatic metric if one can compute it, otherwise human | The survey splits evaluation this way and lists six human criteria (accuracy, relevance, fluency, transparency, safety, human alignment), without a protocol [arxiv:2307.03109 §5] [§5.2]. | | Many generated outputs, and you want a scorer that tracks what users pick | Pairwise picks with ties, then a scorer | The authors report that their scorer predicted held-out user picks better than expert annotators did [arxiv:2305.01569 tab. 1]. Users are self-selected, and the paper does not measure how representative they are [arxiv:2305.01569 §2] [§5]. | | Truthfulness of replies, with a crowd and a fixed pool of claims | Forced choice by several raters per pair | The study used five raters per pair to test whether humans prefer a truthful reply [arxiv:2310.13548 §4.3] [app. D.3]. The raters had no internet access [§4.3]. | | Effects over days in a real setting, with validated questionnaires | Field study with a mixed model | Repeated sessions let a mixed model estimate condition and day effects [arxiv:2402.17937 §2.5] [§3.2.2]. The day effect had 38.7% power, so a null result does not show no change [arxiv:2402.17937 §3.2.2]. | | Expert review of generated text in a high-stakes domain, where you need the error types and not a ranking | Expert Likert ratings with highlights | The authors read their results as a case for physician oversight, since raters found wrong and invented content [arxiv:2212.14882 §5.2]. With one generator and no baseline, the study cannot say which system is better [§3.2]. | | A metric must be validated against people before you rely on its scores | Human ratings as the reference, with agreement reported | The TIFA study's human ratings had alpha 0.67, and the metric correlated with them better than CLIPScore did, on one benchmark [arxiv:2303.11897 §5.1] [tab. 2]. Check the agreement on your own data before trusting the correlation. | | A set-level metric for generated video must be shown to track people better than frame-level scores | Pairwise human comparison, agreement reported per metric | The FVD study ran pairwise comparisons with a "similar quality" option and selected models that tie or spread on the metric being tested [arxiv:1812.01717 §4.3]. The card reports FVD agreeing with raters more often than SSIM and PSNR in its comparisons, on BAIR models only [arxiv:1812.01717 tab. 1] [§4.3]. | | Translation or other high-quality output, and you need to know which errors occur, not only a score | Error-span annotation with severity weights, three raters per segment | The authors argue that listed errors give a base from which several scores can be derived [arxiv:2104.14478 §1]. The weights were set by the authors, and the cost estimate is generic, about USD 1 per segment with three annotators [app. A]. Their simulation puts the budget for a stable system ranking at 951 ratings per system for English-German, a simulated figure [arxiv:2104.14478 tab. 8]. | | Factual errors in generated summaries, with each sentence as the unit and error types wanted | Sentence-level labels with an error typology and three crowd raters | The authors report that 92% of XSum and 43% of CNN/DM summaries have at least one error, and that agreement on the category choice is moderate (kappa 0.39) [arxiv:2104.13346 §4] [§3]. Report the category agreement next to any per-type counts. | | You want to know whether people can detect machine-written text, or to use such a test as an evaluation | Do not use an untrained quick test for current models; add labelled examples if you must use people | The authors argue that untrained raters look at surface features that current models already get right, so the test may measure the wrong thing [arxiv:2107.00061 §5]. The only significant training gain was from worked examples, at 55% overall [arxiv:2107.00061 §3.2]. | | Two versions of one system, the same people in both, with a validated scale | Within-subject questionnaire with a t-test, and report which test and effect size you used | The Pepper study used this design, but its card finds that the paper does not say whether the t-test was paired, and gives no effect size [arxiv:2412.05024 §5.1] [§5.2]. | | A robot or setup compared on a task count plus workload and trust questionnaires | Task outcomes and questionnaires, with groups that differ only in the factor under test | The TELESIM comparison confounds the arm with the country, the gripper and the recruiting route, so its arm ranking does not isolate the arm [arxiv:2410.18727 §IV] [§VI]. | | Several prompted models compared on a small item set by a small rater panel | Per-task-type 1-to-5 averages, with a named test, rater background and blinding stated | The Social-LLaVA study reports averages for three models on 50 items with 15 judges, but gives no test, rater background or statement of whether raters knew the model [arxiv:2501.09024 §IV-B] [tab. 1]. | | A suite of automatic judges across output types must be checked before you rank systems on it | Human check of the judges on sampled outputs, two labels per item with a third on disagreement, agreement reported per modality | The MMMG authors report 94.4% agreement overall, lowest for interleaved audio-text at 91.0%. The check covers 41 of 55 tasks, so validate the rest on your own before relying on their scores [arxiv:2505.17613 §1] [tab. 8]. | | Small annotator pool, or heavy attrition expected | Recruit with margin | The GPT-3 study recruited 100 and analysed 90 [arxiv:2205.11930 app. A.2]. The DALL-E study recruited 60 and kept 29 [arxiv:2205.11930 app. B.2]. | ## Implementation sketch Common core of SPA, in our own words. Each line that matters has a citation. ```pseudocode # Draw m prompts from the task's prior; one output per prompt per system # [arxiv:2205.11930 §5.2 step 1] prompts = sample_prompts(task_prior, m) shown = {sys: [generate(sys, p) for p in prompts] for sys in (X, Y)} # Each annotator sees the same bundle, and is asked both orders # for a percent chance, scaled to 0-1 [arxiv:2205.11930 §5.2 step 2] for a in annotators: p_xy[a] = ask_chance_x_better(a, shown[X], shown[Y]) / 100 p_yx[a] = ask_chance_x_better(a, shown[Y], shown[X]) / 100 # Optional filter: keep annotators whose two answers do not sum above 1.1 # [arxiv:2205.11930 §5.2 step 3] kept = [a for a in annotators if p_xy[a] + p_yx[a] <= 1.1] # Average the per-annotator probabilities and test against chance 0.5 # [arxiv:2205.11930 §5.2 step 4] mean_p = mean(p_xy[a] for a in kept) p_value = two_sided_t_test(p_xy[kept], null_mean=0.5) # Direction: mean above or below 0.5. Strength: the test # [arxiv:2205.11930 §5.1] ``` For pairwise trials feeding a ranking model, the core is a list of records, one per trial, with the two policy ids and an outcome of loss, tie or win. The fit maximises the likelihood of those outcomes [arxiv:2506.18123 alg. 2]. The trial itself is drawn at random from the pool and run blind to the evaluator [arxiv:2506.18123 §3.1]. For error-span scoring, the core is a list of labelled error spans per annotator. Each annotator's score is the weighted sum of at most five target errors, and a segment score is the mean over annotators [arxiv:2104.14478 §3.3] [app. B]. The card has the full pseudocode. Where the variants differ: the standard protocol replaces the two probability questions with one rating per output, then pools the ratings per system. Its inputs need the scale to mean the same thing across annotators, which is the assumption the paper challenges [arxiv:2205.11930 §3]. Settings the authors used, for reference: five prompts per annotator, giving 25 outputs per annotator, read in one batch [arxiv:2205.11930 §6.1] [app. A.1]; generation with temperature 0.9, top-p 1 and at most 1600 tokens [arxiv:2205.11930 app. A.1]; the question asked for a percent chance on a 0-100 scale [arxiv:2205.11930 §5.2 step 2]; the filter threshold τ = 1.1, not applied in the GPT-3 study [arxiv:2205.11930 app. A.2] and applied in the DALL-E study [arxiv:2205.11930 app. B.2]. The study paid USD 5 for about 20 minutes of work [arxiv:2205.11930 app. A.2]. The RoboArena fit ran 60 EM iterations, with a tie-rate start of 0.5 [arxiv:2506.18123 tab. 1] [alg. 2]. ## Pitfalls - **The probability question is noisy.** The two orders do not give exactly complementary answers [arxiv:2205.11930 §5.2 step 2]. Decide before the run whether to apply the τ = 1.1 filter, and report the choice. The DALL-E study found the filter made the two sides more symmetric and increased effect sizes [arxiv:2205.11930 app. B.2]. - **Exposure changes the answer.** An annotator's probability depends on how many outputs they have seen. At zero exposure it is 0.5 [arxiv:2205.11930 §5.1]. In the DALL-E study, one prompt made annotators over-confident, and the gain between 4 and 8 prompts was small [arxiv:2205.11930 §6.2]. The authors used m = 5 for GPT-3 and say the best m for other tasks is not stated [arxiv:2205.11930 §6.1]. - **Use the same prompts for both systems.** Different prompts are allowed, but they make the comparison harder and leave annotators less certain [arxiv:2205.11930 §5.2 step 1]. - **Test details are underspecified.** The paper says two-sided Student's t-test against 0.5, but not which annotators enter each test. The GPT-3 study used one probability per annotator per comparison [arxiv:2205.11930 §5.2 step 4] [app. A.2]. It applied Holm–Bonferroni correction across comparisons; the number of comparisons for the DALL-E study is not stated [arxiv:2205.11930 §6.1] [§6.2]. - **Count annotators, not HITs.** The authors excluded annotators who submitted more than one task, to control how much each one sees [arxiv:2205.11930 app. A.2]. - **Remove length as a cue.** The human text was trimmed to the longest machine output for the same prompt. Do this per prompt, not across the whole set [arxiv:2205.11930 app. A.1]. - **Expected orderings are priors.** The GPT-3 expectations come from earlier work, and the paper treats them as expectations, not ground truth [arxiv:2205.11930 §6.1]. - **Comparisons are not symmetric in effort.** In the GPT-3 study each annotator rated the first output from each writer under the standard protocol, not all 25, so the standard protocol had less within-annotator data [arxiv:2205.11930 app. A.2]. - **The argument rests on assumptions the study does not check.** The critique depends on the rationality axioms and the Likert model. SPA depends on two assumptions, and the experiments do not verify them [arxiv:2205.11930 §5.1]. - **Scope is narrow.** The evidence in the SPA card covers two experiments: GPT-3 stories and DALL-E images. Other tasks are untested [arxiv:2205.11930 limitations]. - **Automatic checks are not human checks.** In the negotiation study, the authors' own evaluation model is a supervised model trained on human-human dialogue, which the paper says may not match human behaviour in human-bot chat [arxiv:1808.09637 §4.4]. Reproducing the human ratings needs crowd workers and an online deployment [arxiv:1808.09637 limitations]. - **Small pairwise studies.** The INSPIRED human study has 28 participants and no reported statistical test [arxiv:2009.14306 §6.3] [app. E]. - **Preference and progress can disagree.** In RoboArena, the preference label disagrees with the progress scores in 11% of trials [arxiv:2506.18123 app. D]. A progress-only ranking is also reported, so pick the signal before the run. - **The protocol and the fit name different outcomes.** RoboArena's protocol text asks for a binary preference [arxiv:2506.18123 §3.1], while the fit takes a tie outcome [alg. 2] [app. B.1]. The card does not say how a tie is recorded by an evaluator, so check the released interface before copying the design. - **Coding is one researcher's judgment.** In the coding study, one of the researchers was the instructor, the sample was one course, and the authors name these as limits [arxiv:2405.17739 limitations]. The authors also say that some students' self-reports are unreliable [arxiv:2405.17739 §4.1]. - **Tie labels need a scoring rule.** Under the Pick-a-Pic metric, a tie label earns partial credit against a non-tie prediction [arxiv:2305.01569 §4 Metric]. Compare numbers only within one metric, and refit the tie threshold on validation for each new scorer [arxiv:2305.01569 §4 Tie Threshold Selection]. - **The human baseline is under-described.** The expert annotators in Pick-a-Pic are friends and colleagues of the authors, and their number and instructions are not stated [arxiv:2305.01569 §4 Baselines]. Treat their 68.0% as a reference point, not a crowd ceiling. - **Ground truth can be wrong.** The sycophancy misconception set may contain factual statements, which the authors admit [arxiv:2310.13548 app. D.1]. Raters judged replies without the internet, and were not the people holding the belief [arxiv:2310.13548 §4.3]. - **Human and model numbers use different pairs.** The sycophancy paper's preference-model result and its human result compare different reply sets, and its sections use different datasets and models [arxiv:2310.13548 §4.1] [§4.2] [§4.3]. Do not read the numbers against each other. - **A survey gives no protocol.** The survey lists factors to set in a human study, but no rater count, scale or agreement threshold [arxiv:2307.03109 §5.2]. - **Small longitudinal studies do not reconcile on counts.** In the social-robot study, the test degrees of freedom imply fewer participants than the 38 completers [arxiv:2402.17937 §3.2.1] [§2.3.3]. Check the analysed n before reusing an effect size. - **Measuring can change the outcome.** Anxiety was measured before and after every session, which may by itself reduce anxiety [arxiv:2402.17937 §4.4]. - **Length can drive the verdict.** The medical study says its longer answers may account for part of the preference [arxiv:2305.09617 §4.2] [§6]. The summarisation study matched lengths by giving the model a 25-word instruction, because it overshot the writers' length target of about 50 words [arxiv:2301.13848 §4.1]. - **Report agreement with the result.** Krippendorff's alpha was 0.07 for overall preference and 0.32 for informativeness, and the authors describe the agreement as moderate. A single preference number hides large differences between annotators [arxiv:2301.13848 §4.2]. - **Offer an exit option only to people.** TIFA's human VQA adds "None of the above", which the models do not get. The human accuracy is therefore not a like-for-like upper bound [arxiv:2303.11897 §5.4] [app. C]. - **Summarise ordinal ratings with medians.** The radiology study reports means and standard deviations of 1-to-5 codes, but medians and quartiles are the safer summary, since the codes are ordinal [arxiv:2212.14882 §3.4]. On the harm statement a high score means low perceived harm, the opposite direction to the other two statements [§3.3]. - **Ties can decide the agreement score.** In the video-metric study's resolution test, a tie counted as a disagreement with FVD [arxiv:1812.01717 §4.3.1] [fig. 5]. Define how ties are scored before you compare agreement numbers. - **Pairwise and individual ratings are not comparable.** The medical study's rubric is not a formally validated instrument. Pairwise and individual evaluation differ on three axes, so their scores cannot be read against each other [arxiv:2305.09617 §3.5] [§6]. - **Released MQM scores have the opposite sign.** The paper's segment score runs from 0 (perfect) upward, but the released average file stores the negative [arxiv:2104.14478 app. B] [code: newstest2020/ende/mqm_newstest2020_ende.avg_seg_scores.tsv#L2]. The repository has no scoring script, and the paper leaves ties in choosing the five most severe errors unspecified [arxiv:2104.14478 §3.3]. Mixing the two conventions reverses the ranking. - **The Major weight is a choice.** The weight 5 came from a resampling check over 1 to 10, and the paper does not specify the procedure or release its code [arxiv:2104.14478 §3.3]. Re-derive it before using the scheme on other data. - **Untrained agreement is very low.** Krippendorff's alpha stayed at 0.11 or less in every training setting, so a single rater's answer is a poor signal [arxiv:2107.00061 §3.2] [tab. 2]. The training gain was significant overall but not within any one domain, and most of it came from stories [arxiv:2107.00061 §3.2]. - **Length cutoffs and copy checks are underspecified.** The detection study does not say whether its 100-word minimum also applies to human texts, and it does not define its overlap measure for regenerating stories [arxiv:2107.00061 §2.2] [§2.2.1]. - **Category labels agree less than the factual label.** Fleiss kappa is 0.58 for factual versus not, but the category choice is weaker. When two annotators agreed on the coreference category, the third chose "Not an error" 69.7% of the time [arxiv:2104.13346 §3] [app. A.5]. Read per-category results with that in mind. - **Dataset size differs from the release.** The paper states about 2250 summaries, but the released file has 2246 records, with 996 XSum records where the paper states 1000 [arxiv:2104.13346 §3] [code: data/human_annotations.json]. Neither the paper nor the README explains the gap. - **Control variables change the correlations.** The paper's partial correlations control for system and dataset, but the released script controls for system only [arxiv:2104.13346 §5.2] [code: evaluation/evaluate.py#L62]. Check which control you use before comparing metric numbers. - **Name the test and the pairing.** The Pepper study names a t-test, reports only p-values, and tests three factors without a stated correction [arxiv:2412.05024 §5.2]. The TELESIM study names no test, normality check or correction for its many comparisons [arxiv:2410.18727 §V-A] [§V-B]. Read its P < 0.1 results as weak. - **Non-significant is not a trend.** The Pepper paper reads discomfort at p = 0.056 as an indication of an effect, although it is above the stated 0.05 level [arxiv:2412.05024 §5.2]. Treat it as a non-significant result. - **Check the sign of the trust scale.** NARS measures negative attitudes toward robots. The TELESIM card notes that the text and the figure caption read its direction differently [arxiv:2410.18727 §V-C] [fig. 12]. Confirm the direction before comparing with other studies. - **Check the arithmetic against the design.** The TELESIM text gives about 40 seconds per tower for a user who built 25 towers, which does not fit a 10-minute run [arxiv:2410.18727 §V-A] [§IV]. Recheck any per-unit figure against the protocol before reusing it. - **Judges need a written protocol.** The Social-LLaVA card notes that its 1-to-5 scale has no published rubric, that the raters' background is not given, and that blinding is not stated [arxiv:2501.09024 §IV-B]. The baselines received three examples and a long prompt, while the proposed model was fine-tuned on about 2K scenarios, so the comparison is asymmetric [arxiv:2501.09024 §III-B] [§IV-B]. - **Agreement figures differ inside one card.** The MMMG abstract gives 94.4% overall, and the per-modality figures in its introduction average to about 93.4%. Its human-agreement table caption gives 0.944, while the average row shows 0.935 for the judge and 0.980 for inter-annotator agreement, and the text gives 97.1% for the latter [arxiv:2505.17613 §1] [§4.1] [tab. 8]. Quote the figure and its row, not the headline alone. - **Judge agreement covers a subset of tasks.** Human validation reaches 41 of the 55 tasks. Do not carry the overall figure to tasks outside that set [arxiv:2505.17613 §1]. ## Contradictions - **Competence with regional speech.** The Pepper study finds hardly any competence difference between the two versions, but the prior work it cites reported a possible drop for non-standard speech [arxiv:2412.05024 §5.3]. That prior work is not carded here, so this page cannot check it. Card: [Talking Like One of Us](/papers/2412.05024). - **Trust across countries.** The TELESIM study reports more trust in robots in Japan, and says this goes against an earlier cross-country study [arxiv:2410.18727 §V-C]. The earlier study is not carded here, so this page cannot check it. Card: [Breaking Down the Barriers](/papers/2410.18727). - **Human-likeness in the negotiation study.** The text says the supervised word model matches dialogue length and utility better, while the caption of the same table says the act-based models match the human statistics better across all metrics [arxiv:1808.09637 §4.4] [tab. 7]. The table's length column supports the text. Card: [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637). - **Harm share in the radiology study.** The text says "about half" of the radiologists found critical statements [arxiv:2212.14882 §5.1.3]. The table gives 36% listing harmful conclusions [tab. 3], and the paper does not explain the gap. Card: [ChatGPT Makes Medicine Easy to Swallow](/papers/2212.14882). - **Within the coding study.** The text and the tables disagree on some per-student labels and on one correlation [arxiv:2405.17739 limitations]. The card lists them [arxiv:2405.17739 limitations]. Card: [The Widening Gap](/papers/2405.17739). - **Within the FRANK card.** The text swaps the error shares of BERTSum and BART relative to the table [arxiv:2104.13346 §4] [tab. 3]. The text also gives BERTScore P a Spearman of 0.35, where the table gives 0.24 [arxiv:2104.13346 §5.3] [tab. 2]. Card: [FRANK](/papers/2104.13346). The other cards do not compare the same quantity, so no other contradiction is known among them. Raters also differ in expertise: the translation study uses professional translators, and the detection study uses untrained crowd workers. The detection card reports that untrained raters were at chance, and the translation card cites earlier work that professional translators can tell human from machine translation [arxiv:2104.14478 §4.1]. These are different tasks, and neither paper tests the other's setting. ## Papers - [Towards Accurate Generative Models of Video: A New Metric & Challenges](/papers/1812.01717): proposes FVD for generated video and checks it against pairwise human comparisons on BAIR models, with SSIM and PSNR as the comparison metrics. - [The Authenticity Gap in Human Evaluation](/papers/2205.11930): argues that Likert averages can reverse true preferences; proposes system-level probabilistic assessment with GPT-3 and DALL-E experiments. - [Decoupling Strategy and Generation in Negotiation Dialogues](/papers/1808.09637): rates bot human-likeness on crowd chats, and compares supervised and reinforcement-learned negotiation models. - [INSPIRED: Toward Sociable Recommendation Dialog Systems](/papers/2009.14306): uses a pairwise human comparison with a "can't tell" option and five-point ratings for a movie-recommendation chatbot. - [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](/papers/2506.18123): ranks robot policies from double-blind pairwise trials with a task-aware Bradley-Terry model. - [The Widening Gap](/papers/2405.17739): codes think-aloud notes from observed novice programming sessions with generative AI tools. - [Benchmarking Large Language Models for News Summarization](/papers/2301.13848): rates ten LLMs' summaries on faithfulness, coherence and relevance, and runs a blinded pairwise test against freelance-writer summaries. - [Towards Expert-Level Medical Question Answering with Large Language Models](/papers/2305.09617): physicians and lay raters score long answers on a rubric, and physicians rank Med-PaLM 2 and physician answers pairwise on nine axes. - [Pick-a-Pic](/papers/2305.01569): collects pairwise image picks, with ties, from open-platform users, and trains a scorer on them. - [A Survey on Evaluation of Large Language Models](/papers/2307.03109): classifies LLM evaluation, including the automatic and human families; no experiments. - [Towards Understanding Sycophancy in Language Models](/papers/2310.13548): uses five crowd raters per pair to compare truthful and sycophantic replies. - [LLM-Powered Socially Assistive Robot-Delivered CBT Exercises](/papers/2402.17937): a 15-day study with questionnaires and a mixed-effects analysis. - [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](/papers/2212.14882): 15 radiologists rate Likert statements and highlights on one generator's simplified radiology reports. - [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](/papers/2303.11897): uses 1-to-5 human ratings and a second human question-answering protocol to check a text-to-image metric. - [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](/papers/2104.14478): professional translators mark error spans with severity, giving an MQM score that reorders the WMT20 systems. - [All That's "Human" Is Not Gold: Evaluating Human Evaluation of Generated Text](/papers/2107.00061): untrained crowd raters are at chance on GPT-3 text; short training helps only a little, and the authors advise against small untrained studies. - [Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan](/papers/2410.18727): 74 non-experts teleoperate three arms through a digital twin, with tower counts, NASA-TLX and NARS; the arm and the country are confounded. - [Talking Like One of Us: Effects of Using Regional Language in a Humanoid Social Robot](/papers/2412.05024): 17 participants hear one robot dialogue in standard and regional German and rate it on RoSAS; warmth differs, and the test type is not stated. - [Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces](/papers/2501.09024): fifteen judges score a fine-tuned vision-language model's answers on a 1-to-5 scale against two prompted models, with no reported statistical test. - [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](/papers/2104.13346): crowd workers label each summary sentence as factual or not and tag its error type, on about 2250 CNN/DM and XSum summaries; checks factuality metrics against those labels. - [MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613): 55 generation tasks scored by automatic judges, checked against human annotators on 41 of them, with agreement reported per modality.