Human evaluation

Overview

Human ratings are often treated as the reference for judging generated text, dialogue or robot behaviour. The question is how to turn many individual judgments into a verdict on which system is better, or into a finding about why something happened. Twenty-one cards on this wiki use human judgments in different ways. One argues that the usual average of Likert ratings can reverse the true preference 2205.11930 §3. Others rate single outputs for human-likeness 1808.09637 §4.4, ask for pairwise preferences 2009.14306 §6.3, rank policies from double-blind pairwise trials 2506.18123 §3.1, code what people said during a session 2405.17739 §3.4, collect pairwise image picks from users at scale 2305.01569 §2, have crowd workers choose between a truthful and a sycophantic reply 2310.13548 §4.3, or follow users' outcomes over 15 days with questionnaires 2402.17937 §2.5. Three robot cards add three more designs: a task-outcome study with workload and trust questionnaires across three arms 2410.18727 §IV, a within-subject rating of a robot's regional and standard speech 2412.05024 §5.1, and a 1-to-5 rating of a vision-language model's answers by fifteen judges 2501.09024 §IV-B. A summarisation card has annotators rate LLM summaries on several aspects, then run blinded pairwise tests 2301.13848 §3.1 §4.2, and a medical QA card has physicians rank long answers pairwise on nine axes 2305.09617 §3.5. A survey card lists the human criteria used for LLMs and says when a criterion needs people at all 2307.03109 §5. Three more cards differ in purpose. A radiology study has 15 radiologists rate ChatGPT rewrites of medical reports for errors and harm 2212.14882 §3.3, and a text-to-image metric is checked against human ratings of how well each image matches its prompt 2303.11897 §5.1, and a video-metric card runs pairwise human comparisons to check a set-level metric against frame-level ones 1812.01717 §4.3. Three cards cover error labelling and detection. In translation, professional raters mark each error span with a category and a severity, and the marks become a weighted segment score 2104.14478 §3.3. In text generation, untrained crowd workers say whether a short text was written by a person or by GPT-3 or GPT-2, and a few training methods are tested against that baseline 2107.00061 §2.1 §3.1. In summarisation, crowd workers label each sentence of a summary as factual or not, and tag any error type 2104.13346 §3.

Most variants share a skeleton: a fixed sample of outputs from each system, judged by several annotators, and a statistical test or model at the end. They differ in what the annotator is asked and in how the answers are combined. For an LLM used as the annotator, see LLM-as-a-judge.

All numbers on this page are the authors' own and come from studies on specific tasks, several of them small. No card tests one variant against another on the same task, except where a card reports its own baseline.

Variants

Absolute ratings averaged per system (standard Likert protocol)

Each annotator rates single outputs on a Likert scale. The ratings are averaged per system and the systems are ranked by that average 2205.11930 §1. The authors say the average estimates expected utility only if annotators satisfy the von Neumann–Morgenstern axioms and agree on their best and worst outcomes, so that ratings can be pooled 2205.11930 §2.4 §3. On GPT-3 story generation, this protocol recovered 2 of 5 expected preferences 2205.11930 tab. 1. Card: The Authenticity Gap in Human Evaluation.

The same kind of rating appears in dialogue work. Crowd workers chatted with a bot or a human and rated the partner's human-likeness on a 1-to-5 scale. Human partners scored 4.3 on the new Craigslist data 1808.09637 §4.4 tab. 7. Card: Decoupling Strategy and Generation in Negotiation Dialogues. A five-point rating of fluency, consistency, naturalness, persuasiveness and engagingness was also collected with 25 participants per model 2009.14306 app. E tab. 10. Card: INSPIRED: Toward Sociable Recommendation Dialog Systems.

A multi-aspect version scores each summary for faithfulness as a yes-or-no judgment, and for coherence and relevance on a 1-to-5 scale. Three annotators rate each summary and the ratings are averaged 2301.13848 §3.1. Ten LLMs were scored this way on two news datasets, next to the existing reference summaries and two fine-tuned systems 2301.13848 tab. 2. Card: Benchmarking Large Language Models for News Summarization.

A small robot-language study uses the same 1-to-5 scoring with a panel of fifteen judges. Each answer to 50 scene questions is scored, and the scores are averaged per task type for three models. The two baselines are prompted only, and the proposed model is fine-tuned 2501.09024 §IV-B tab. 1. The authors call the gaps significant, but the card finds no test, variance or interval in the paper 2501.09024 §IV-B. Card: Social-LLaVA.

Repaired absolute protocol

The same per-output ratings, with three changes: a continuous scale instead of Likert points, maximum and minimum reference outcomes shown to annotators, and no outcome that makes a system unusable 2205.11930 §4. The paper notes that a protocol with several of these features has been used for machine translation since 2017, and that such a protocol cannot be applied to open-ended tasks 2205.11930 §4. No experiment in the card tests this variant. Card: The Authenticity Gap in Human Evaluation.

System-level probabilistic assessment (SPA)

Each annotator reads a fixed sample of outputs from both systems and states the percent chance, from 0 to 100, that system X is the better writer. The probabilities are averaged across annotators and tested against 0.5 2205.11930 §5.2. Because each answer lies between 0 and 1, the answers can be averaged directly, with no pooling of scale values. The authors say SPA needs two assumptions: stated preferences are unbiased on average, and preferences are complementary (P[X ≻ Y] + P[Y ≻ X] = 1), with indifference at 0.5 [arxiv:2205.11930 §5.1]. The standard protocol needs the stronger conditions listed above; SPA does not 2205.11930 §5.1.

Evidence from the authors' GPT-3 study, with the expected ordering from their prior 2205.11930 §6.1:

Source: 2205.11930 tab. 1. SPA matched the expected direction in all five rows, and the standard protocol in two. The authors read the power of the tests as near 1 for both protocols at the observed effects, so they take the standard protocol's failure to be a measurement problem, not a lack of precision 2205.11930 §6.3. In a DALL-E image study, scrambling the prompt gave significantly worse images at α = 0.01 after correction, as expected 2205.11930 §6.2. Card: The Authenticity Gap in Human Evaluation.

Pairwise preference with a "can't tell" option

Each annotator chats with or reads two systems and picks the better one on a named aspect, with an option to say they cannot tell. The INSPIRED study used this on five aspects with 28 participants, each chatting with both models two or three times 2009.14306 §6.3. The text reports that the strategy model was preferred on all five aspects, with the largest gap on engagingness, but gives no counts or test 2009.14306 §6.3 fig. 5. Card: INSPIRED.

The summarisation study used a blinded pairwise test with a tie option, for overall preference and for informativeness. Each annotator judged the same 100 pairs 2301.13848 §4.2. Overall preference came out equal in aggregate between a zero-shot instruction-tuned model and freelance writers. The more abstractive summary was judged more informative 51.1% of the time, which the authors do not treat as significant 2301.13848 §4.2. Card: Benchmarking Large Language Models for News Summarization.

Multi-axis pairwise ranking by domain experts

Physicians compare two answers to the same question and pick the better one, or a tie, on each of nine axes. The axes cover medical consensus, reading comprehension, knowledge recall, reasoning, unwanted and omitted content, demographic bias, and two harm measures 2305.09617 §3.5. Med-PaLM 2 answers were preferred to physician-written answers on eight of the nine axes, at p < 0.001. The exception was inaccurate or irrelevant content 2305.09617 §4.2 fig. 5. The paper uses bootstrap intervals and two-tailed permutation tests, each with 10,000 iterations, and blocks the permutation test by answer when an answer has several ratings 2305.09617 §3.5. The number of raters per pair in this pairwise task is not stated 2305.09617 §3.5. Card: Towards Expert-Level Medical Question Answering with Large Language Models.

Double-blind pairwise trials with a ranking model

Evaluators run head-to-head trials of two policies on tasks they choose. They do not know which policy is which. Each trial gives a progress score, a binary preference and a free-text reason 2506.18123 §3.1. The records are fit with a Bradley-Terry model with latent task buckets, by EM, with a Davidson tie term 2506.18123 §3.2 alg. 2 app. B.1. The authors report that this ranking is closer to an exhaustive oracle than the standard DROID protocol, and that it is stable after about 100 comparisons 2506.18123 §5.2 §5.3. Card: RoboArena. For the ranking model alone, see Bradley-Terry model.

Qualitative coding of observed sessions

Researchers watch a person work, take notes line by line, and tag each line with a codebook. Two coders first code the same sessions, then discuss disagreements 2405.17739 §3.4. This answers why people struggle, not which system is better. The study of 21 novice programmers reached Cohen's kappa 0.74 on the first sessions and reported three new difficulty codes 2405.17739 §3.4 tab. 2. Card: The Widening Gap.

Pairwise picks from open-platform users, with ties

Users type a prompt, see two generated images, and pick one or call a tie. The rejected image is replaced and the loop repeats for that prompt 2305.01569 §2. The authors chose the two-image, tie-allowed design over four images without ties and over two without ties, after a pilot on engagement and inter-rater agreement 2305.01569 §2. The released data has 583,747 training examples 2305.01569 §2. A scorer trained on the picks predicted held-out user choices at 70.5%, against 68.0% for expert annotators 2305.01569 tab. 1. The authors then use this scorer to rank images from other models 2305.01569 §5. Card: Pick-a-Pic.

Forced choice by crowd workers on truthful versus sycophantic replies

For each pair of replies, five crowd workers pick the one they prefer, without internet access 2310.13548 §4.3 app. D.3. The study has 1,330 comparisons over 266 misconceptions 2310.13548 app. D.3 §4.3. The text says truthful replies were usually preferred, less reliably at higher difficulty, and gives no number 2310.13548 §4.3.1 fig. 7b. Card: Towards Understanding Sycophancy.

Field study with self-report instruments and a mixed model

Participants use one condition for many days. Each session has a before and an after anxiety survey, and the study tracks a distress instrument on days 1, 8 and 15 2402.17937 §2.5. The analysis uses a mixed-effects model with pre-session anxiety as a covariate 2402.17937 §2.6.3 §3.2.2. The study had 38 completers across a robot, a chatbot and a worksheet group, over 15 days 2402.17937 §2.3.3. Distress fell in the robot and worksheet groups and not in the chatbot group 2402.17937 §3.2.1. Post-session anxiety was about 6 points lower in the robot and chatbot groups than in the worksheet group 2402.17937 tab. 4. Card: LLM-Powered Socially Assistive Robot-Delivered CBT Exercises.

Within-subject questionnaire comparison with t-tests

The same participants experience both versions of one system, and each rates the result on a validated scale after every run. The Pepper study had 17 adults hear the same dialogue in High German and in Low German. After each dialogue they rated the 18 items of the Robotic Social Attributes Scale on a 5-point scale 2412.05024 §5.1. For each of three factors, a t-test compared the two versions at significance level 0.05 2412.05024 §5.2. Warmth was higher for the Low German version (p = 0.027). Competence (p = 0.378) and discomfort (p = 0.056) were not significant 2412.05024 §5.2. The paper does not say whether the test was paired, although the design is within-subject 2412.05024 §5.1 §5.2. Card: Talking Like One of Us.

Task outcomes and questionnaires across robots and groups

Non-experts steer one robot arm through a digital copy of the robot, for ten minutes, to stack three cubes into a tower. Afterwards they answer a single-ease question, the raw NASA-TLX for workload, and five items from the NARS S1 subscale, which the authors use as a proxy for trust 2410.18727 §IV §II-C. The study had 74 people from the UK and Japan and three arms. The UR5e built a mean of 10.50 towers per user, against 3.25 for Baxter and 1.03 for the UR3, and it had the lowest mental workload 2410.18727 §V-A §V-B. The arm, its gripper, its controller and the country all differed together, so the comparison cannot separate the arm from the country 2410.18727 §IV-A §VI. Card: Breaking Down the Barriers.

Expert Likert ratings with highlighted errors, one generator

Radiologists read three original reports and one randomly chosen simplified version of each. They rate three statements on a 5-point scale: the simplified report is factually correct, it includes the relevant information, and it could lead a patient to wrong conclusions and harm 2212.14882 §3.3. Each statement has a follow-up where the rater highlights wrong text, lists missing information or lists harmful conclusions §3.3. The authors report medians, quartiles and ranges per statement, then group the free-text answers by content §3.4 §4.2. The study had 15 raters and 45 simplified reports from one generator and no baseline, so it describes one system rather than ranking several §3.2 §3.3. Medians were 2 (agree) for correctness and completeness and 4 (disagree) for harm, and 51% of raters highlighted at least one incorrect passage tab. 1 tab. 3. Card: ChatGPT Makes Medicine Easy to Swallow.

Human ratings as the reference for an automatic metric

Here the human judgments are the ground truth for checking a metric, not the final result. Annotators rate each generated image from 1 to 5 for how well it matches its prompt. The TIFA study used 20 graduate students, 800 images from five generators on 160 prompts, and two annotators per image, with Krippendorff's alpha 0.67 2303.11897 §5.1 app. C tab. 2. The metric's Spearman correlation with these ratings was 59.7 with mPLUG as the VQA model, against 33.2 for CLIPScore 2303.11897 tab. 2. A second protocol has annotators answer the metric's own multiple-choice questions on the generated images. It adds a "None of the above" option, uses two annotators per question with a third on disagreement, and reports alpha 0.88 2303.11897 §5.4 app. C. Card: TIFA.

A video metric gets the same treatment in a different form. Raters compare two generated videos and pick the better one or call them similar quality. Model pairs are chosen so that they tie on one metric, or spread across its range, and each row of the agreement table holds one metric fixed 1812.01717 §4.3 tab. 1. With SSIM held equal, FVD agreed with raters on 74.9% of comparisons, and with PSNR held equal on 81.0% 1812.01717 tab. 1. The human study used BAIR models from four families, with 10 models per condition [arxiv:1812.01717 §4.3]. Card: Towards Accurate Generative Models of Video.

Factuality metrics are checked against sentence-level labels in the same way. FRANK correlates each metric with the summary-level human score, after removing the effect of the system that wrote the summary 2104.13346 §5.2. On all data, the best partial Pearson correlation is 0.27, for BERTScore P, and the best Spearman is 0.30, for FactCC 2104.13346 tab. 2. Entailment-based metrics do best on CNN/DM, and their correlation on XSum is about a quarter of that 2104.13346 §5.3 fig. 4. Card: FRANK.

A benchmark for generated images, audio and mixed outputs uses people to check its automatic judges, not to rank systems. Annotators label sampled outputs, and their labels are compared with the judges' decisions. Two annotators label each question and a third settles disagreements. Agreement is reported by modality, from 91.0% for interleaved audio-text to 96.6% for interleaved image-text, and 94.4% overall. The pool was 20 graduate students 2505.17613 §4.1 tab. 8. The human check covers 41 tasks and 978 instructions, so the other tasks are not covered by it 2505.17613 §1. Card: MMMG.

Error-span annotation with severity weights (MQM)

Professional annotators read a whole document, highlight each error span, and give each error a category and a severity. The paper's scheme has three severities (Major, Minor, Neutral) and error categories grouped under accuracy, fluency, terminology, style and locale 2104.14478 §3.3 app. B. The segment score is a weighted sum. Major errors weigh 5, Minor 1, Neutral 0, and Non-translation 25 for a segment too garbled to annotate further. At most the five most severe target errors per annotator count 2104.14478 tab. 1 app. B. Three professional raters annotate each segment 2104.14478 tab. 2. The authors report that every human translation ranks above every machine system under MQM and pSQM, in both language pairs 2104.14478 §4.1. The English-German data set has 1,418 segments and the Chinese-English one 2,000 2104.14478 §3.4. Card: Experts, Errors, and Context.

Sentence-level factuality labels with an error typology, crowd majority

Crowd workers read each sentence of a generated summary in the context of the full summary, and say whether it is factual. If not, they pick one or more error types from a seven-type typology, plus "Other" and "Not an error" 2104.13346 §2 §3. Three annotators label each item. A sentence counts as wrong when a majority marks it, and a summary counts as wrong when any sentence does 2104.13346 §4. The authors ran a pilot that rated whole summaries on a Likert scale and dropped it, because they found it less precise and less consistent 2104.13346 §3, footnote. Workers needed at least 85% on a qualification test. During the task, workers were checked with entity questions and with altered sentences, and those below 50% on the altered sentences were blocked 2104.13346 §3 app. A.6. The labelled set has about 2250 summaries from nine systems on two news datasets. The authors report that 92% of XSum summaries and 43% of CNN/DM summaries have at least one error 2104.13346 §4. The overall 60% figure could not be reproduced from the per-system rows, so treat it as unverified 2104.13346 tab. 3. Fleiss kappa is 0.58 for factual versus not, and 0.39 for the category choice when all three annotators agree a sentence is not factual 2104.13346 §3. Card: FRANK.

Untrained crowd detection of machine text, with optional training

Crowd workers on Amazon Mechanical Turk read five texts from one domain and one model, a mix of human and machine text, and label each text on a four-point scale 2107.00061 §2.1 §2.3. The authors compare this untrained task with three short trainings: instructions, worked examples, and a comparison of paired texts 2107.00061 §3.1. Only the worked examples gave a significant overall gain, to 55% against 50% 2107.00061 §3.2. Untrained raters were at chance on GPT-3 text (49.9%) and above chance on GPT-2 text (57.9%) [arxiv:2107.00061 §2.4] tab. 1. Card: All That's "Human" Is Not Gold.

When to use which

Implementation sketch

Common core of SPA, in our own words. Each line that matters has a citation.

# Draw m prompts from the task's prior; one output per prompt per system
# [arxiv:2205.11930 §5.2 step 1]
prompts = sample_prompts(task_prior, m)
shown = {sys: [generate(sys, p) for p in prompts] for sys in (X, Y)}

# Each annotator sees the same bundle, and is asked both orders
# for a percent chance, scaled to 0-1  [arxiv:2205.11930 §5.2 step 2]
for a in annotators:
    p_xy[a] = ask_chance_x_better(a, shown[X], shown[Y]) / 100
    p_yx[a] = ask_chance_x_better(a, shown[Y], shown[X]) / 100

# Optional filter: keep annotators whose two answers do not sum above 1.1
# [arxiv:2205.11930 §5.2 step 3]
kept = [a for a in annotators if p_xy[a] + p_yx[a] <= 1.1]

# Average the per-annotator probabilities and test against chance 0.5
# [arxiv:2205.11930 §5.2 step 4]
mean_p = mean(p_xy[a] for a in kept)
p_value = two_sided_t_test(p_xy[kept], null_mean=0.5)

# Direction: mean above or below 0.5. Strength: the test
# [arxiv:2205.11930 §5.1]

For pairwise trials feeding a ranking model, the core is a list of records, one per trial, with the two policy ids and an outcome of loss, tie or win. The fit maximises the likelihood of those outcomes 2506.18123 alg. 2. The trial itself is drawn at random from the pool and run blind to the evaluator 2506.18123 §3.1.

For error-span scoring, the core is a list of labelled error spans per annotator. Each annotator's score is the weighted sum of at most five target errors, and a segment score is the mean over annotators 2104.14478 §3.3 app. B. The card has the full pseudocode.

Where the variants differ: the standard protocol replaces the two probability questions with one rating per output, then pools the ratings per system. Its inputs need the scale to mean the same thing across annotators, which is the assumption the paper challenges 2205.11930 §3.

Settings the authors used, for reference: five prompts per annotator, giving 25 outputs per annotator, read in one batch 2205.11930 §6.1 app. A.1; generation with temperature 0.9, top-p 1 and at most 1600 tokens [arxiv:2205.11930 app. A.1]; the question asked for a percent chance on a 0-100 scale 2205.11930 §5.2 step 2; the filter threshold τ = 1.1, not applied in the GPT-3 study 2205.11930 app. A.2 and applied in the DALL-E study 2205.11930 app. B.2. The study paid USD 5 for about 20 minutes of work 2205.11930 app. A.2. The RoboArena fit ran 60 EM iterations, with a tie-rate start of 0.5 2506.18123 tab. 1 alg. 2.

Pitfalls

• The probability question is noisy. The two orders do not give exactly complementary answers 2205.11930 §5.2 step 2. Decide before the run whether to apply the τ = 1.1 filter, and report the choice. The DALL-E study found the filter made the two sides more symmetric and increased effect sizes 2205.11930 app. B.2.

• Exposure changes the answer. An annotator's probability depends on how many outputs they have seen. At zero exposure it is 0.5 2205.11930 §5.1. In the DALL-E study, one prompt made annotators over-confident, and the gain between 4 and 8 prompts was small 2205.11930 §6.2. The authors used m = 5 for GPT-3 and say the best m for other tasks is not stated 2205.11930 §6.1.

• Use the same prompts for both systems. Different prompts are allowed, but they make the comparison harder and leave annotators less certain 2205.11930 §5.2 step 1.

• Test details are underspecified. The paper says two-sided Student's t-test against 0.5, but not which annotators enter each test. The GPT-3 study used one probability per annotator per comparison 2205.11930 §5.2 step 4 app. A.2. It applied Holm–Bonferroni correction across comparisons; the number of comparisons for the DALL-E study is not stated 2205.11930 §6.1 §6.2.

• Count annotators, not HITs. The authors excluded annotators who submitted more than one task, to control how much each one sees [arxiv:2205.11930 app. A.2].

• Remove length as a cue. The human text was trimmed to the longest machine output for the same prompt. Do this per prompt, not across the whole set 2205.11930 app. A.1.

• Expected orderings are priors. The GPT-3 expectations come from earlier work, and the paper treats them as expectations, not ground truth 2205.11930 §6.1.

• Comparisons are not symmetric in effort. In the GPT-3 study each annotator rated the first output from each writer under the standard protocol, not all 25, so the standard protocol had less within-annotator data [arxiv:2205.11930 app. A.2].

• The argument rests on assumptions the study does not check. The critique depends on the rationality axioms and the Likert model. SPA depends on two assumptions, and the experiments do not verify them 2205.11930 §5.1.

• Scope is narrow. The evidence in the SPA card covers two experiments: GPT-3 stories and DALL-E images. Other tasks are untested 2205.11930 limitations.

• Automatic checks are not human checks. In the negotiation study, the authors' own evaluation model is a supervised model trained on human-human dialogue, which the paper says may not match human behaviour in human-bot chat 1808.09637 §4.4. Reproducing the human ratings needs crowd workers and an online deployment 1808.09637 limitations.

• Small pairwise studies. The INSPIRED human study has 28 participants and no reported statistical test 2009.14306 §6.3 app. E.

• Preference and progress can disagree. In RoboArena, the preference label disagrees with the progress scores in 11% of trials 2506.18123 app. D. A progress-only ranking is also reported, so pick the signal before the run.

• The protocol and the fit name different outcomes. RoboArena's protocol text asks for a binary preference 2506.18123 §3.1, while the fit takes a tie outcome alg. 2 app. B.1. The card does not say how a tie is recorded by an evaluator, so check the released interface before copying the design.

• Coding is one researcher's judgment. In the coding study, one of the researchers was the instructor, the sample was one course, and the authors name these as limits 2405.17739 limitations. The authors also say that some students' self-reports are unreliable 2405.17739 §4.1.

• Tie labels need a scoring rule. Under the Pick-a-Pic metric, a tie label earns partial credit against a non-tie prediction 2305.01569 §4 Metric. Compare numbers only within one metric, and refit the tie threshold on validation for each new scorer 2305.01569 §4 Tie Threshold Selection.

• The human baseline is under-described. The expert annotators in Pick-a-Pic are friends and colleagues of the authors, and their number and instructions are not stated 2305.01569 §4 Baselines. Treat their 68.0% as a reference point, not a crowd ceiling.

• Ground truth can be wrong. The sycophancy misconception set may contain factual statements, which the authors admit 2310.13548 app. D.1. Raters judged replies without the internet, and were not the people holding the belief 2310.13548 §4.3.

• Human and model numbers use different pairs. The sycophancy paper's preference-model result and its human result compare different reply sets, and its sections use different datasets and models 2310.13548 §4.1 §4.2 §4.3. Do not read the numbers against each other.

• A survey gives no protocol. The survey lists factors to set in a human study, but no rater count, scale or agreement threshold 2307.03109 §5.2.

• Small longitudinal studies do not reconcile on counts. In the social-robot study, the test degrees of freedom imply fewer participants than the 38 completers 2402.17937 §3.2.1 §2.3.3. Check the analysed n before reusing an effect size.

• Measuring can change the outcome. Anxiety was measured before and after every session, which may by itself reduce anxiety 2402.17937 §4.4.

• Length can drive the verdict. The medical study says its longer answers may account for part of the preference 2305.09617 §4.2 §6. The summarisation study matched lengths by giving the model a 25-word instruction, because it overshot the writers' length target of about 50 words 2301.13848 §4.1.

• Report agreement with the result. Krippendorff's alpha was 0.07 for overall preference and 0.32 for informativeness, and the authors describe the agreement as moderate. A single preference number hides large differences between annotators 2301.13848 §4.2.

• Offer an exit option only to people. TIFA's human VQA adds "None of the above", which the models do not get. The human accuracy is therefore not a like-for-like upper bound 2303.11897 §5.4 app. C.

• Summarise ordinal ratings with medians. The radiology study reports means and standard deviations of 1-to-5 codes, but medians and quartiles are the safer summary, since the codes are ordinal 2212.14882 §3.4. On the harm statement a high score means low perceived harm, the opposite direction to the other two statements §3.3.

• Ties can decide the agreement score. In the video-metric study's resolution test, a tie counted as a disagreement with FVD 1812.01717 §4.3.1 fig. 5. Define how ties are scored before you compare agreement numbers.

• Pairwise and individual ratings are not comparable. The medical study's rubric is not a formally validated instrument. Pairwise and individual evaluation differ on three axes, so their scores cannot be read against each other 2305.09617 §3.5 §6.

• Released MQM scores have the opposite sign. The paper's segment score runs from 0 (perfect) upward, but the released average file stores the negative 2104.14478 app. B mqm_newstest2020_ende.avg_seg_scores.tsv. The repository has no scoring script, and the paper leaves ties in choosing the five most severe errors unspecified 2104.14478 §3.3. Mixing the two conventions reverses the ranking.

• The Major weight is a choice. The weight 5 came from a resampling check over 1 to 10, and the paper does not specify the procedure or release its code 2104.14478 §3.3. Re-derive it before using the scheme on other data.

• Untrained agreement is very low. Krippendorff's alpha stayed at 0.11 or less in every training setting, so a single rater's answer is a poor signal 2107.00061 §3.2 tab. 2. The training gain was significant overall but not within any one domain, and most of it came from stories 2107.00061 §3.2.

• Length cutoffs and copy checks are underspecified. The detection study does not say whether its 100-word minimum also applies to human texts, and it does not define its overlap measure for regenerating stories 2107.00061 §2.2 §2.2.1.

• Category labels agree less than the factual label. Fleiss kappa is 0.58 for factual versus not, but the category choice is weaker. When two annotators agreed on the coreference category, the third chose "Not an error" 69.7% of the time 2104.13346 §3 app. A.5. Read per-category results with that in mind.

• Dataset size differs from the release. The paper states about 2250 summaries, but the released file has 2246 records, with 996 XSum records where the paper states 1000 2104.13346 §3 human_annotations.json. Neither the paper nor the README explains the gap.

• Control variables change the correlations. The paper's partial correlations control for system and dataset, but the released script controls for system only 2104.13346 §5.2 evaluate.py. Check which control you use before comparing metric numbers.

• Name the test and the pairing. The Pepper study names a t-test, reports only p-values, and tests three factors without a stated correction 2412.05024 §5.2. The TELESIM study names no test, normality check or correction for its many comparisons 2410.18727 §V-A §V-B. Read its P < 0.1 results as weak.

• Non-significant is not a trend. The Pepper paper reads discomfort at p = 0.056 as an indication of an effect, although it is above the stated 0.05 level 2412.05024 §5.2. Treat it as a non-significant result.

• Check the sign of the trust scale. NARS measures negative attitudes toward robots. The TELESIM card notes that the text and the figure caption read its direction differently 2410.18727 §V-C fig. 12. Confirm the direction before comparing with other studies.

• Check the arithmetic against the design. The TELESIM text gives about 40 seconds per tower for a user who built 25 towers, which does not fit a 10-minute run 2410.18727 §V-A §IV. Recheck any per-unit figure against the protocol before reusing it.

• Judges need a written protocol. The Social-LLaVA card notes that its 1-to-5 scale has no published rubric, that the raters' background is not given, and that blinding is not stated 2501.09024 §IV-B. The baselines received three examples and a long prompt, while the proposed model was fine-tuned on about 2K scenarios, so the comparison is asymmetric 2501.09024 §III-B §IV-B.

• Agreement figures differ inside one card. The MMMG abstract gives 94.4% overall, and the per-modality figures in its introduction average to about 93.4%. Its human-agreement table caption gives 0.944, while the average row shows 0.935 for the judge and 0.980 for inter-annotator agreement, and the text gives 97.1% for the latter 2505.17613 §1 §4.1 tab. 8. Quote the figure and its row, not the headline alone.

• Judge agreement covers a subset of tasks. Human validation reaches 41 of the 55 tasks. Do not carry the overall figure to tasks outside that set 2505.17613 §1.

Contradictions

• Competence with regional speech. The Pepper study finds hardly any competence difference between the two versions, but the prior work it cites reported a possible drop for non-standard speech 2412.05024 §5.3. That prior work is not carded here, so this page cannot check it. Card: Talking Like One of Us.
• Trust across countries. The TELESIM study reports more trust in robots in Japan, and says this goes against an earlier cross-country study 2410.18727 §V-C. The earlier study is not carded here, so this page cannot check it. Card: Breaking Down the Barriers.
• Human-likeness in the negotiation study. The text says the supervised word model matches dialogue length and utility better, while the caption of the same table says the act-based models match the human statistics better across all metrics 1808.09637 §4.4 tab. 7. The table's length column supports the text. Card: Decoupling Strategy and Generation in Negotiation Dialogues.
• Harm share in the radiology study. The text says "about half" of the radiologists found critical statements 2212.14882 §5.1.3. The table gives 36% listing harmful conclusions tab. 3, and the paper does not explain the gap. Card: ChatGPT Makes Medicine Easy to Swallow.
• Within the coding study. The text and the tables disagree on some per-student labels and on one correlation 2405.17739 limitations. The card lists them 2405.17739 limitations. Card: The Widening Gap.
• Within the FRANK card. The text swaps the error shares of BERTSum and BART relative to the table 2104.13346 §4 tab. 3. The text also gives BERTScore P a Spearman of 0.35, where the table gives 0.24 2104.13346 §5.3 tab. 2. Card: FRANK.

The other cards do not compare the same quantity, so no other contradiction is known among them. Raters also differ in expertise: the translation study uses professional translators, and the detection study uses untrained crowd workers. The detection card reports that untrained raters were at chance, and the translation card cites earlier work that professional translators can tell human from machine translation 2104.14478 §4.1. These are different tasks, and neither paper tests the other's setting.

Papers

• Towards Accurate Generative Models of Video: A New Metric & Challenges: proposes FVD for generated video and checks it against pairwise human comparisons on BAIR models, with SSIM and PSNR as the comparison metrics.
• The Authenticity Gap in Human Evaluation: argues that Likert averages can reverse true preferences; proposes system-level probabilistic assessment with GPT-3 and DALL-E experiments.
• Decoupling Strategy and Generation in Negotiation Dialogues: rates bot human-likeness on crowd chats, and compares supervised and reinforcement-learned negotiation models.
• INSPIRED: Toward Sociable Recommendation Dialog Systems: uses a pairwise human comparison with a "can't tell" option and five-point ratings for a movie-recommendation chatbot.
• RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies: ranks robot policies from double-blind pairwise trials with a task-aware Bradley-Terry model.
• The Widening Gap: codes think-aloud notes from observed novice programming sessions with generative AI tools.
• Benchmarking Large Language Models for News Summarization: rates ten LLMs' summaries on faithfulness, coherence and relevance, and runs a blinded pairwise test against freelance-writer summaries.
• Towards Expert-Level Medical Question Answering with Large Language Models: physicians and lay raters score long answers on a rubric, and physicians rank Med-PaLM 2 and physician answers pairwise on nine axes.
• Pick-a-Pic: collects pairwise image picks, with ties, from open-platform users, and trains a scorer on them.
• A Survey on Evaluation of Large Language Models: classifies LLM evaluation, including the automatic and human families; no experiments.
• Towards Understanding Sycophancy in Language Models: uses five crowd raters per pair to compare truthful and sycophantic replies.
• LLM-Powered Socially Assistive Robot-Delivered CBT Exercises: a 15-day study with questionnaires and a mixed-effects analysis.
• ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports: 15 radiologists rate Likert statements and highlights on one generator's simplified radiology reports.
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering: uses 1-to-5 human ratings and a second human question-answering protocol to check a text-to-image metric.
• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation: professional translators mark error spans with severity, giving an MQM score that reorders the WMT20 systems.
• All That's "Human" Is Not Gold: Evaluating Human Evaluation of Generated Text: untrained crowd raters are at chance on GPT-3 text; short training helps only a little, and the authors advise against small untrained studies.
• Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan: 74 non-experts teleoperate three arms through a digital twin, with tower counts, NASA-TLX and NARS; the arm and the country are confounded.
• Talking Like One of Us: Effects of Using Regional Language in a Humanoid Social Robot: 17 participants hear one robot dialogue in standard and regional German and rate it on RoSAS; warmth differs, and the test type is not stated.
• Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces: fifteen judges score a fine-tuned vision-language model's answers on a 1-to-5 scale against two prompted models, with no reported statistical test.
• Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics: crowd workers label each summary sentence as factual or not and tag its error type, on about 2250 CNN/DM and XSum summaries; checks factuality metrics against those labels.
• MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation: 55 generation tasks scored by automatic judges, checked against human annotators on 41 of them, with agreement reported per modality.