LLM evaluation

Overview

A reported score for a language model depends on more than the model. It depends on the prompt wording, the answer format, the decoding settings, the model snapshot, the metric, the ground truth, and, for open-ended output, the judge. The papers below each apply a different evaluation method, and each states a different set of these details. The shared invariant is that a number is only comparable to another if the setup behind it is known. The ChatGPT study ties its results to the API snapshot used in one week in June 2023 2302.06476 Limitations. Several later papers show the same dependence from other sides: the format of a prompt can move accuracy by many points 2310.11324 §4.2, and the decoding mode can decide whether a result is sampled at all 2307.15343 Pitfalls. An average can also hide per-item instability: a model may flip its answer across equivalent prompts while the dataset mean barely moves 2410.12405 §2.1.

Variants

Zero-shot benchmark run with one fixed prompt

Each test item is sent as one prompt: a short task instruction plus the input, with no demonstrations 2302.06476 §3. The study covers 20 datasets in 7 categories, comparing ChatGPT with text-davinci-003 under the same setup 2302.06476 §4.1. Reasoning tasks add a two-stage zero-shot chain-of-thought step 2302.06476 §3. The results are mixed. ChatGPT beats the older model on five of six arithmetic sets without the chain and on all six with it 2302.06476 tab. 2. On GSM8K it reaches 78.9 against 59.5 for text-davinci-003 2302.06476 tab. 2. It trails on CommonsenseQA (73.7 against 74.9) and on all three summarisation ROUGE scores [arxiv:2302.06476 tab. 3] 2302.06476 tab. 9. The paper's own comparison concludes that it underperforms earlier fine-tuned methods in most cases [arxiv:2302.06476 §4.3]. Card: Is ChatGPT a General-Purpose Natural Language Processing Task Solver?.

Two details limit how far such a run can be trusted. Some test or validation sets are small (CB has 56 items, COPA has 100 validation items) and the paper gives no confidence intervals 2302.06476 tab. 1. Off-format replies, such as "It is unclear" for a yes/no question, account for part of the gap the paper reports, and how they were scored is not stated [arxiv:2302.06476 §4.2.4].

Many analytical tasks, each scored against its best system

Some studies run one chat model over a long task list and compare each task with the best system for that task, not with one shared baseline. ChatGPT was run on 25 analytical NLP tasks, 23 in English and 2 in Polish, most of them classification 2302.10724 §4. Averaged over the tasks, it scores 56.51 against 73.71 for the reference, a loss of 25.50%. The loss is 52.59% on the emotion tasks and 12.90% on the semantic ones 2302.10724 tab. 2. The reference rows mix the authors' reruns with numbers from source papers that used other setups and splits, so the averages mix both 2302.10724 §5.3 2302.10724 tab. 2. The model version and sampling settings are not stated, answers were parsed by hand, and answers outside the label set were dropped before scoring 2302.10724 §5.2 2302.10724 §5.3. Part of each per-task loss is built in, since difficulty is 100 minus the reference score 2302.10724 §6.2. Personalised in-context prompts did not reliably help: GoEmoPer1 and GoEmoPer2 score below GoEmoPer0 2302.10724 tab. 2. Card: ChatGPT: Jack of all trades, master of none.

One protocol across settings, against fine-tuned baselines

Some papers run the same model under several settings and compare it with a fine-tuned system from the literature. The results depend on which baseline is used.

The biomedical benchmark runs four LLMs on 12 test sets under zero-shot, static few-shot, dynamic nearest-neighbour few-shot and LoRA fine-tuning 2305.16326 p. 4 2305.16326 tab. 3. The paper reports that the fine-tuned baseline beats the best zero- or few-shot LLM on 10 of 12 datasets, and its macro-average is 0.6536 against 0.4862 for the best few-shot LLM 2305.16326 p. 6 2305.16326 tab. 3. GPT-4 with five static examples leads the prompted settings, at 0.7651 accuracy on MedQA against a 0.4195 baseline 2305.16326 tab. 3. The baselines are published numbers and were not re-run, and the macro-average mixes ROUGE-L, F1 and accuracy 2305.16326 p. 14 2305.16326 p. 12. Card: Benchmarking large language models for biomedical natural language processing applications and recommendations.

On relation extraction, few-shot GPT-3 comes close to the supervised systems, but only after humans re-judge exact-match errors. About half of the false positives on ADE and CoNLL04 are judged correct by workers 2305.05003 §3.2. The in-context GPT-3 F1 is 76.53 on CoNLL04 and 82.66 on ADE, against 71.48 to 75.44 and 79.22 to 83.15 for the fully supervised rows 2305.05003 tab. 2. The paper says that exact matching underestimates generated output 2305.05003 §2, and it does not give the rule that applies the human judgements to precision and recall 2305.05003 §3.2. Card: Revisiting Relation Extraction in the era of Large Language Models.

Generated training data can be judged against public instruction-tuned models. The Self-Instruct study fine-tunes GPT-3 on its own generated instructions and tests the result zero-shot on 119 unseen SuperNI tasks, with 100 instances each, scored by ROUGE-L 2212.10560 §4.3 2212.10560 tab. 3. The untuned model scores 6.8, the tuned one 39.9, and the InstructGPT-001 and T0 rows 40.8 and 33.1 2212.10560 tab. 3. The paper compares only with text-davinci-001 among the InstructGPT models, because newer engines may have seen the test set 2212.10560 §4.2. Its human check uses 252 author-written instructions with one output each, rated on four levels by experts 2212.10560 §4.4. The reported agreement is 0.57 in the text and 0.58 in the appendix 2212.10560 §4.4 2212.10560 app. B.2. Counting the top two levels as valid, the model is about 5% behind InstructGPT-001 2212.10560 §4.4. Evaluation is greedy with no stop sequences and a 1024-token cap, so a model that keeps writing is scored on everything it produces up to that cap 2212.10560 tab. 4 2212.10560 §4.3. The generated data is also noisy: one author judged 200 sampled items, and only 54% had all fields valid 2212.10560 tab. 2. Card: Self-Instruct: Aligning Language Models with Self-Generated Instructions.

Zero-shot ChatGPT has been run across languages in the same way. The authors give one plain prompt per example, with no worked examples, on seven tasks: part-of-speech tagging, named-entity recognition, relation extraction, natural language inference, question answering, commonsense reasoning and summarisation 2304.05613 §3 2304.05613 §4. The method names 34 languages and the introduction names 37, so the per-task tables are the safer count 2304.05613 §3 2304.05613 §1. On tagging, the average with English descriptions is 84.5 against 79.3 for XLM-R. On named-entity recognition, the macro-F1 average is 29.9 against 88.4 for the supervised baseline, and on natural language inference 57.0 against 87.1 for mT5-XXL 2304.05613 §4 2304.05613 §5 2304.05613 §7. The paper's headline is that the model is significantly behind supervised systems on most tasks, and that English descriptions usually do better 2304.05613 §11. The tagging win over XLM-R is 13 of 17 languages only if the better of the two prompts is taken per language. With English descriptions alone it is 12 2304.05613 §4. The model snapshot is not pinned, and responses were collected between 1 March and 5 April 2023 2304.05613 §3. Card: ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning.

Likelihood-ranked choice on a base model

Base models are often scored without a chat template. The LLaMA report ranks each answer option by its likelihood under the model and takes the most likely one 2302.13971 §3. For most datasets the likelihood is normalised by the option's number of characters. For OpenBookQA and BoolQ it is divided by the likelihood of the option after an "Answer:" prompt, following GPT-3 2302.13971 §3. Question answering uses greedy output, cut at the first line break, final dot or comma, and is scored by exact match [arxiv:2302.13971 app. A]. The paper reports LLaMA-13B above GPT-3 on most common-sense tasks, although it is about ten times smaller 2302.13971 §3.1 [arxiv:2302.13971 tab. 3]. TriviaQA is scored on the filtered dev set, while GPT-3 and PaLM report on the unfiltered test set, so the two are not comparable [arxiv:2302.13971 app. A] 2302.13971 tab. 5. Most baseline numbers are copied from other papers and were not re-run 2302.13971 §3.1. Card: LLaMA: Open and Efficient Foundation Language Models.

The same scoring idea applies to reasoning items with no examples. Analogy puzzles were given to text-davinci-003 with no demonstrations, and each answer choice was appended to the prompt and scored by its mean token log-probability 2212.09196 §4.2. Temperature was 0 and each item was run once, and the paper reports no run-to-run variance for the main results [arxiv:2212.09196 Pitfalls]. Against average undergraduate accuracy, the model's odds ratio is 1.88 for generated answers on digit matrices and 6.27 for the eight-choice version [arxiv:2212.09196 §2.1] fig. 3. On story analogies it is weaker than the students, with an odds ratio of 0.37, and its far-analogy results are not significantly above chance 2212.09196 §2.4 fig. 8. The paper's figures carry most of the per-condition values, so exact numbers need the authors' data files 2212.09196 Pitfalls. Scoring by the mean rather than the sum of token log-probabilities changes how choices of different length compare, and the paper does not discuss that choice 2212.09196 Pitfalls. Card: Emergent Analogical Reasoning in Large Language Models.

Sweeps over settings: shots, reasoning mode, format and language

Some results change with a setting the benchmark author chose, so one number hides a range. Four cards show how to report that range.

MMLU-Pro extends the four-option MMLU to ten options, with harder reasoning items and removed noisy ones. The paper evaluates with 5-shot chain of thought and reports GPT-4o at 72.6% overall 2406.01574 §5.1 [arxiv:2406.01574 tab. 2]. Its reasoning mode matters more than on MMLU: for GPT-4o, chain of thought adds 19.1 points on MMLU-Pro and 1.5 on MMLU 2406.01574 tab. 3. Prompt changes move MMLU scores by about 4 to 5 points, with a peak of 10.98, and MMLU-Pro scores by about 2, with a maximum of 3.74 2406.01574 §6.3.

Format spread goes further. The authors generate few-shot prompts that differ only in formatting, keep their meaning, and measure the gap between the best and worst format for each model and task 2310.11324 §3. The median spread is 7.5 accuracy points across tasks for LLaMA-2 and Falcon [arxiv:2310.11324 §4.2], and the largest is 76 points for LLaMA-2-13B 2310.11324 §1. Testing every format is costly, so the paper uses Thompson sampling to find the best and worst format within a fixed budget, and reports that it lands within 1 point of the true spread at its largest budget, where the UCB comparison is within 11 points 2310.11324 §4.5. Card: Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design.

Belebele tests reading comprehension on 900 questions in 122 language variants 2308.16884 §3.5. Its tables set zero-shot, few-shot and fine-tuned models next to each other. English is far easier than the average: LLaMA 2 70B scores 90.9 on English with five shots and 48.0 on average across all variants 2308.16884 tab. 2. Translating the test into English first changes the picture for LLaMA 2-chat 70B: 57.1 on average with translate-test, against 44.1 when the questions stay in their own language 2308.16884 tab. 3. The paper flags one zero-shot result as unfair, because that model was fine-tuned on translation data, and does not analyse it 2308.16884 tab. 2. Card: The Belebele Benchmark.

The o1 system card runs a zero-shot multilingual MMLU on human-translated test sets in 14 languages plus English 2412.16720 §6. Each question is sent alone, and the answer letter is taken from the reply. o1 scores 0.8900 in Arabic and 0.7538 in Yoruba, against 0.8155 and 0.6195 for GPT-4o [arxiv:2412.16720 tab. 17]. The paper's harness and README give a different o1 figure for the English MMLU, so cite the paper's table for these numbers [arxiv:2412.16720 Paper vs code]. Card: OpenAI o1 System Card.

Sensitivity can also be measured per item instead of per setting. PromptSensiScore averages, over the instances of a benchmark, the mean absolute gap in score between the prompt variants of one instance 2410.12405 §2.2 2410.12405 eq. 1 2410.12405 eq. 2. The objective sets use 12 variants per item and greedy decoding 2410.12405 §3.1. Llama3-70B-Instruct is reported as the most robust model, and few-shot examples lower the score for all four Qwen1.5 models tested, with the largest drop from zero to one shot 2410.12405 §3.3 2410.12405 §3.4. A score below 0.1 is read as high robustness, which the authors offer as a rule of thumb 2410.12405 §2.2. The formula divides by the number of unordered prompt pairs, which the text does not say 2410.12405 eq. 1. Card: ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs.

Translation shows the same effect. Three GPT models were scored on 18 directions from WMT21 and WMT22 test sets, with 0, 1 or 5 example pairs in the prompt 2302.09210 §2.1 2302.09210 §3.1. How the pairs are chosen is a setting too: random pairs, pairs drawn from a high-quality pool, and pairs retrieved by similarity 2302.09210 §3.1. For text-davinci-003, the best few-shot setting beats zero-shot on 17 of the 18 directions and ties on ZH-EN 2302.09210 tab. 3 2302.09210 tab. 4. More than five shots gave no gain in preliminary runs, and retrieved pairs gave no gain over the quality-pool pairs when translating from English 2302.09210 §3.1 2302.09210 §3.3. Document-level prompting is a different trade: on DE-EN, a window of 16 sentences moves COMET-22 from 84.8 to 85.2 and cuts GPT requests from 1,984 to 310 2302.09210 tab. 5. BLEU and chrF fall for every GPT setting while COMET does not, so the metric changes the reading 2302.09210 §3.2 2302.09210 §3.3. The paper states no sampling settings for these API calls 2302.09210 §2.3. Card: How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.

A translate-then-classify pipeline adds a step that is itself unmeasured. A study of depression-severity labels on Reddit posts runs GPT-3.5-turbo at temperature 0 twice: once on the English posts, and once on Greek translations that the same model produced from them 2410.12985 §4 [arxiv:2410.12985 fig. 1]. The English macro F1 is 0.17 and the Greek macro F1 is 0.16, and the English run is the only baseline 2410.12985 tab. 2. The paper reports no translation metric and checks only two examples by eye, so the Greek score mixes classifier and translation errors 2410.12985 Pitfalls. Whether the classification prompt was also translated is not stated [arxiv:2410.12985 §4]. Card: Leveraging LLMs for Translating and Classifying Mental Health Data.

A minimal set of trials can show how a probe's result moves with the number of in-context examples. Theory-of-mind probes of GPT-3 use k-shot likelihood scoring, with k from 0 to 35 on a social-commonsense set and from 2 to 24 on a story set with true- and false-belief questions 2210.13312 §2.1 2210.13312 §3.1. The best social-commonsense result is about 55% for the largest model at 35 shots, and gains beyond 10 shots are about one point 2210.13312 §2.2. On the story set, accuracy on mental-state questions peaks near 60%, and the true-belief curve rises and then falls at k = 4, which the authors link to recency bias 2210.13312 §3.2 2210.13312 app. B. Factual questions in the same set reach 90 to 100% 2210.13312 §3.2. The shot choice is random and seeded, and the paper reports no spread across seeds [arxiv:2210.13312 Pitfalls]. The authors add that solving these tasks is necessary but not sufficient evidence of theory of mind 2210.13312 §5. Card: Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.

Reporting rule that follows: publish the setting next to each number, and where a method can reach a score by a path the others cannot, show it separately 2308.16884 tab. 2 2406.01574 tab. 3.

Complexity curves and where reasoning collapses

Scaling-curve reading asks whether a result on small models predicts what larger models will do. The survey defines an ability as emergent if it is absent in smaller models and present in larger ones, which makes it a shape on a scaling curve: near random up to some scale, then clearly above random 2206.07682 §2. The authors plot against training FLOPs and list emergence thresholds for the tasks and methods they study in one table [arxiv:2206.07682 tab. 1]. The points are judged by hand, not by a numeric threshold, and the labelling used two co-authors who kept only the labels they agreed on 2206.07682 app. A.3.

The main tool for the metric question is to plot a smoother measure next to accuracy. For six BIG-Bench tasks, cross-entropy of the target answer improves with scale before accuracy leaves chance, and the authors call this the case where improvement is hidden by the metric 2206.07682 app. A.1. Partial- credit metrics such as BLEU and ROUGE still show the jump on the generative tasks checked, so the metric alone does not explain it [arxiv:2206.07682 app. A.2]. Scale is also not the only factor: PaLM 62B is above random on 14 BIG-Bench tasks where the larger LaMDA 137B and GPT-3 175B are near random 2206.07682 §5.2. Card: Emergent Abilities of Large Language Models.

A controllable input size gives a cleaner curve. The Illusion of Thinking study uses four puzzles (Tower of Hanoi, Checker Jumping, River Crossing and Blocks World), each with a reasoning model and its non-thinking twin, 25 samples per point, and matched token budgets 2506.06941 §4.1. The authors describe three regimes: at low complexity the non-thinking model is as good or better with fewer tokens; at medium complexity the reasoning model leads; at high complexity both fall near zero 2506.06941 §4.2. For Claude 3.7 Sonnet with thinking, the Tower of Hanoi collapse falls around 7 to 8 disks, or about 100 to 200 moves 2506.06941 app. A.1. Giving the algorithm in the prompt leaves the collapse point largely unchanged for Hanoi and Checker Jumping 2506.06941 §4.4. Sampling matters: at temperature 0, DeepSeek-R1 collapses earlier on Hanoi (N = 8) than at temperature 1 2506.06941 app. A.6. The paper's boat-capacity rule and failure positions are not consistent across sections 2506.06941 §3.1 2506.06941 app. A.1, so use the numbers as approximate. Card: The Illusion of Thinking.

Novel or minimally edited probes

Here the evaluator writes tasks that are unlikely to be in the training data, then changes them to check whether the model is reasoning or recalling 2303.12712 §1.1. The authors say standard held-out benchmarks are not enough for this model, partly because it appears to know the BIG-bench canary GUID, a sign that benchmark text was seen in training 2303.12712 §1.1. For memorisation, they take a question that GPT-4 got right and text-davinci-003 got wrong, then re-sample its numbers. GPT-4 scores 75.2% on the re-sampled versions and text-davinci-003 scores 0.2% [arxiv:2303.12712 fig. 4.2]. The questions were picked because of that difference, so the gap is conditional on the selection; this is our reading, not a claim of the paper.

The same design shows how sensitive results are to the prompt. For the expression a·b + c·d, GPT-4 answers directly with 58% accuracy when the numbers are 0 to 9 and 0% for 99 to 199, but reaches 100% for 1 to 40 when asked to write intermediate steps first 2303.12712 §8.2. Explanations are also tested separately: a model's own explanation can match its output without predicting its behaviour on other inputs, so the paper checks explanations with editing and override experiments [arxiv:2303.12712 §6.2]. The authors call the whole approach subjective and informal 2303.12712 §1.1. Card: Sparks of Artificial General Intelligence: Early experiments with GPT-4.

A failure archive gives hard cases without a rate. One author collected ChatGPT mistakes from public posts and their own prompts, sorted them into eleven categories from spatial reasoning and logic to arithmetic, facts, bias, code, grammar, self-description and ethics, and said the categories are arbitrary and not exhaustive 2302.03494 §1 2302.03494 §2. The archive reports no counts, no sampling rule and no baseline, so it cannot say how often a failure occurs 2302.03494 Pitfalls. Some items show the answer changing with the wording: a reworded family-count question gets 8 where the original gets the correct 5 2302.03494 §4. A rephrased request that gets past a refusal is one of the items, so the archive should be used as a safety test set for such cases 2302.03494 §2.10 2302.03494 Pitfalls. Card: A Categorical Archive of ChatGPT Failures.

A minimal edit can be a stronger probe than a new task. The Theory-of-Mind critique takes two published vignettes and changes one detail at a time, such as making the container transparent or having a trusted friend report the contents 2302.08399 §2.1 2302.08399 §2.2. With GPT-3.5, the belief completion moves to the label's content: chocolate receives 95% for a transparent bag, where the baseline reported in that paper gives 99% 2302.08399 §2.1.1 2302.08399 §2.1. The author argues that one failure on a trivial variant should count for more than many passes 2302.08399 §1 2302.08399 §3. Limits: one model, one prompt per variant, no repeated runs and no statistics 2302.08399 Pitfalls. A double space before one sentence changed a result, so the exact strings matter 2302.08399 §2.1.1. Card: Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks.

Classic psychology experiments run as trial-by-trial tasks

Some studies take experiments from cognitive psychology and run them on a model one trial at a time, with the model's earlier answers and the feedback it received kept in the prompt 2206.14576 Method. The horizon bandit and two-step tasks are generated by code, the gamble problems come from an existing benchmark, and human data come from the cited literature 2206.14576 Results. In the study of GPT-3, the model's regret on gambles is worse than humans' [arxiv:2206.14576 Results] fig. 2B. On the horizon bandit its overall regret is lower than humans', but the paper finds no directed exploration 2206.14576 Results fig. 3D. In the two-step task the stay pattern is model-based 2206.14576 Results fig. 4C. The headline result is that the model fails the causal cask task [arxiv:2206.14576 Discussion]. The vignettes are well-known problems that may be in the training data, a single deterministic answer per prompt is reported, and the paper gives no run-to-run variance 2206.14576 Pitfalls. The repository's temperature settings do not match the text 2206.14576 Paper vs code. Card: Using cognitive psychology to understand GPT-3.

Long-context, frontier and calibration-aware benchmarks

Some benchmarks are built to stay hard for current models, and the paper must report a human reference or a calibration measure with the accuracy.

EgoSchema asks multiple-choice questions about long egocentric video clips, with 5,063 instances and five options each 2308.09126 app. Datasheet. Zero-shot models score between about 20% and 32% against a chance level of 20%, with the best open model at 32.1% for InternVideo with 90 frames [arxiv:2308.09126 §4.2]. Human accuracy is 75.0% with no time limit 2308.09126 §4.2. The paper reports a median temporal certificate of about 100 seconds, estimated on 100 clips 2308.09126 §4.1. Card: EgoSchema.

Humanity's Last Exam collects 2,500 closed-ended expert questions, about 14% with images, and filters out questions that frontier models solved 2501.14249 §4.2. Eight models score between 2.7% and 13.4% accuracy, and their RMS calibration error stays between 73% and 89% 2501.14249 tab. 1. The authors say that the filter causes part of the low score, and that residual accuracy comes from noise and guessing 2501.14249 §4.2. Accuracy is judged by o3-mini 2501.14249 §4.2. Card: Humanity's Last Exam.

A case study covers many tasks with few items each. The o1 evaluation runs o1-preview on 27 tasks with author-written prompts, and passes 10 of 12 LeetCode problems in three attempts, 83.3% 2409.18486 tab. 2. On radiology reports its ROUGE-1 is 0.3019, above gpt-4-turbo at 0.2999, but its ROUGE-2 is 0.0448, below gpt-4-turbo at 0.0656 2409.18486 tab. 3. Most tasks have no baseline, and the linked repository holds no evaluation code 2409.18486 §1.4, so treat these results as a description of one model on one set of prompts. Card: Evaluation of OpenAI o1.

A domain study can follow the same pattern. The finance overview runs GPT-4 under zero-shot or one-shot prompts on six public datasets, with no baseline model 2401.11641 §4 2401.11641 tab. 2. It reports 0.78 accuracy on Financial PhraseBank and 0.81 entity F1 on named-entity recognition, but 0.53 on stock movement 2401.11641 tab. 2. The stock labels leave the band between -0.5% and +0.55% unassigned, and the paper does not say how those cases were scored 2401.11641 §4.1. The prompt text, model snapshot and decoding settings are not stated, so the numbers cannot be rebuilt from the paper 2401.11641 §4.2. Card: Revolutionizing Finance with LLMs: An Overview of Applications and Insights.

LLM as judge: pairwise picks and rubric grading

For open-ended answers with a reference, the paper asks GPT-4 to compare its own answer with an answer from GPT-3 against a gold statement, writes a short explanation for each, and must pick exactly one 2303.12712 §7.2.2. On TruthfulQA, the judge picked GPT-4's answer 87.76% of the time and GPT-3's 11.01%, while human reviewers picked GPT-4's answer 47.61% of the time and chose "both" for 23.29% 2303.12712 tab. 6. The judge and the humans agree on 50.8% of items 2303.12712 §7.2.2. The 89.83% figure for humans comes from removing the "neither" and "both" options and is not a new labelling 2303.12712 tab. 6. The authors did not check agreement between their two human reviewers 2303.12712 §7.2.2. Because the judge compares its own answer with a rival, the comparison is not neutral 2303.12712 §7.2.2. Card: Sparks of Artificial General Intelligence: Early experiments with GPT-4.

A survey of LLM judges lists the choices that move a judge's output: the prompt, the model, post-processing, and the reliability tricks of multi-round and multi-model voting 2411.15594 Method. Its small experiment on six LLMs finds that GPT-4-turbo is the least biased of the set, and that five-round majority voting with swapped answer positions helps most 2411.15594 §4.4. Alignment with human labels is close across models: 61.5% for GPT-4-turbo, 54.7% for GPT-3.5-turbo, and 61.7% for o3-mini on the same human-preference set 2411.15594 §4.4. Card: A Survey on LLM-as-a-Judge.

Judge design also moves three-way support labels (supported, partly supported, unsupported) against a reference. One study compares four single-call designs: splitting the candidate answer into claims, a holistic rubric with a completeness item, splitting the reference into claims, and a bidirectional mix of both 2603.28005 §3.2. On ASQA the holistic rubric beats candidate-side splitting for all four judges tested, by 7.25 to 15.5 points [arxiv:2603.28005 §4.1] 2603.28005 tab. 2. Reference-side splitting is the most accurate ASQA design for the three primary judges, at 0.975, 0.9325 and 0.8675 for Opus 4.6, GPT-4.1 and Gemini Flash Lite, and it uses 28 to 29% more tokens than the holistic rubric 2603.28005 §4.2 2603.28005 tab. 2. The labels are constructed from the datasets, and the paper says they measure agreement with a strict completeness standard, not factual accuracy 2603.28005 §3.1 2603.28005 Pitfalls. On 60 rows labelled by one author, the ranking of candidate-side and holistic designs reverses, by one or two rows per judge, and the reference-side design comes last 2603.28005 §6.1. The author's labels have no agreement check 2603.28005 Pitfalls, so the reversal is weak evidence on its own. Card: Atomic and Holistic LLM Judges for Reference-Grounded Support Labels.

A rubric judge can be checked against teachers on exam answers. GAOKAO-Bench has 2,811 questions from national college entrance exams from 2010 to 2022, of which 1,781 are objective and 1,030 subjective 2305.12474 §1 2305.12474 tab. 3. For subjective answers, teachers write marking criteria that split each answer into scoring points. GPT-4-turbo then grades zero-shot at temperature 0 2305.12474 §3.4. Question-level Spearman correlation with the human scores is 0.825 to 0.864 across five models, and Kendall's tau is 0.685 to 0.740 2305.12474 tab. 4. Without the marking criteria, GPT-4-0613 falls from 0.854 to 0.820 Spearman 2305.12474 tab. 6. Converted totals differ from the human totals by under 2% for sciences and about 5% for humanities 2305.12474 §4.4. The correlations are question-level, not total-score agreement 2305.12474 Pitfalls. Card: Evaluating the Performance of Large Language Models on GAOKAO Benchmark.

GEMBA applies the same idea to machine translation. It scores each translated segment with one zero-shot prompt to a GPT model, on a 0 to 100 direct-assessment scale, with or without the human reference, and averages the segment scores per system 2302.14520 §2 2302.14520 §2.2. System pairs are compared by pairwise accuracy on WMT22 MQM for en-de, en-ru and zh-en [arxiv:2302.14520 §3.1] 2302.14520 §3.2. With GPT-4 and the reference, the accuracy is 89.8%, above MetricX XXL at 85.0%. Without the reference it is 87.6%, against 78.8% for COMETKiwi 2302.14520 tab. 2. Other prompt styles reach 91.2%, namely stars and classes without a reference, so the headline prompt is one choice among several 2302.14520 tab. 3. The smallest models were close to random, and GPT-2 and Ada gave no usable scores 2302.14520 §4.3. If a reply is not a valid number, the paper queries again at a higher temperature, without giving the schedule 2302.14520 §2.2. Card: Large Language Models Are State-of-the-Art Evaluators of Translation Quality.

A trained pairwise ranker is a second kind of judge, and it can be checked against a judge model. PairRanker reads the source with two candidate outputs at once and scores which one is better. Doing this for every pair gives an order over N candidates 2306.02561 §3.2 2306.02561 §3.3. Its training labels come from BARTScore against a reference answer, not from a language model 2306.02561 §5.1. The paper's headline measure is a different thing: GPT-Rank, in which ChatGPT judges every candidate pair and a candidate's rank is its number of wins. On the 5,000-example MixInstruct test split, the fused output reaches GPT-Rank 3.01, against 3.90 for the best single model. The ChatGPT version used as judge is not stated 2306.02561 tab. 2 2306.02561 app. E. Because the ranker and the judge use different labels, the GPT-Rank result is an independent check of the ranker's order. Card: LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion.

LLM rating on a Likert scale, checked against human raters

This variant asks the model for a score instead of a choice. The rater gets the same instruction, text and 5-point question as the human raters, answers in free text, and a parser takes the first number from the answer [arxiv:2305.01937 §2.2] 2305.01937 app. D.2.1. The authors draw three samples per question for T0 and InstructGPT and one answer for ChatGPT, which has no sampling control 2305.01937 §3.2. On story generation, text-davinci-003 and ChatGPT give human-written stories a higher mean than GPT-2 stories on all four attributes. The paper reports the gaps for text-davinci-003 as significant 2305.01937 §3.3 2305.01937 tab. 1. T0 and text-curie-001 show no clear preference 2305.01937 tab. 1. Story-level agreement with the teacher average is positive but modest, with Kendall's tau between 0.14 and 0.43 2305.01937 tab. 2. Changing the instruction to a persona or an explanation request, or lowering the temperature, moves the absolute scores but keeps the human-over-GPT-2 order 2305.01937 tab. 3. Card: Can Large Language Models Be an Alternative to Human Evaluations?.

Human preference protocols

Human raters remain the reference for many tasks, and two cards question how their answers are aggregated.

For news summaries, annotators see the outputs of three systems for one article and pick the most and the least preferred 2209.12356 §3.1. GPT-3 with a sentence-count prompt is the majority-best system on 58% of CNN articles, against 36% for BRIO, and on 57% of BBC articles, against 20% for BRIO [arxiv:2209.12356 tab. 3]. Automatic metrics rank the same outputs lower: on CNN, ROUGE-1 is 31.86 for GPT-3 against 38.49 for BRIO 2209.12356 tab. 4. The authors take the disagreement as a reason not to trust reference-based scores for prompted systems 2209.12356 §4. Annotators were given no definition of quality, and the paper does not report a value for its agreement statistic 2209.12356 §3.1 2209.12356 §3.2. Card: News Summarization and Evaluation in the Era of GPT-3.

The authenticity-gap paper argues that averaging Likert ratings can reverse the true preference when two systems are close, because the scale is ordinal and annotators' magnitudes are not comparable 2205.11930 §3. Its proposed protocol asks each annotator for the probability that they prefer system X to system Y after reading a shared sample of outputs, then averages those probabilities and tests them against 0.5 2205.11930 §5.2. On GPT-3 story generation with 90 analysed annotators, the authors report that this recovers 5 of 5 expected preferences, against 2 of 5 for the standard protocol 2205.11930 §6.1 2205.11930 tab. 1. The method says whether a preference exists, not how large it is 2205.11930 Limitations. Card: The Authenticity Gap in Human Evaluation.

Summaries can also be compared with a stronger reference. Human raters scored ten LLMs on CNN/DM and XSUM for faithfulness, coherence and relevance, and the paper finds that instruction tuning, not model size, decides zero-shot quality 2301.13848 §3.2 2301.13848 tab. 2. The existing reference summaries score 0.37 for XSUM faithfulness, against 0.97 for zero-shot Davinci Instruct v2, so scores measured against those references are unreliable 2301.13848 tab. 2 2301.13848 §3.2. In a blinded pairwise test, freelance-writer summaries and the model's summaries were judged about equal overall. The model's summaries were more extractive, with coverage of 0.92 against 0.81 2301.13848 §4.2. Annotator agreement is low, with Krippendorff's alpha of 0.07 for overall preference 2301.13848 §4.2. Card: Benchmarking Large Language Models for News Summarization.

Expert raters can also rate rewritten text on fixed statements. Fifteen radiologists each rated three ChatGPT rewrites of fictitious radiology reports, 45 in all, on 5-point scales for factual correctness, completeness and potential harm 2212.14882 §3.3 2212.14882 tab. 1. The medians were 2 for correctness, 2 for completeness and 4 for harm 2212.14882 tab. 1. The harm statement says a reader would draw wrong conclusions, so a median of 4 (disagree) means low perceived harm, while a median of 2 on correctness means agreement. The two should not be compared directly 2212.14882 Pitfalls. In the free-text follow-ups, 51% of raters answered the question about wrong passages and 36% the one about harmful conclusions 2212.14882 tab. 3. The paper defines completeness and harm, but not factual correctness 2212.14882 §3.3. Card: ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports.

Long medical answers need more than one score. MultiMedQA pairs six existing question sets with 3,375 consumer questions, and clinicians grade each long answer on twelve items, covering agreement with medical consensus, possible harm, missing content and possible bias for a demographic group 2212.13138 §3.1 2212.13138 §3.2. Each answer has one rater, and the main study uses nine clinicians and 140 questions 2212.13138 §4.5. The paper reports that 61.9% of Flan-PaLM answers agree with scientific consensus, against 92.6% for Med-PaLM (92.9% in the text), and that Med-PaLM is worse on inappropriate or incorrect content, at 18.7% against 16.1% 2212.13138 app. A.7. The authors describe the axes as subjective and only pilot-validated 2212.13138 §6.3. Card: Large Language Models Encode Clinical Knowledge.

Untrained raters are a weak check on fluent text. Crowd workers on Amazon Mechanical Turk judged whether stories, news articles and recipes were written by a person or by GPT-3 or GPT-2 2107.00061 §2.1. Without training they were at chance on GPT-3 text, at 49.9% against 50% 2107.00061 §2.4 2107.00061 tab. 1. Three short training conditions were tried. Only the one that showed labelled examples gave a significant overall gain, to 55%, and no single domain gain was significant 2107.00061 §3.2 [arxiv:2107.00061 tab. 2]. Agreement between raters stayed low, with Krippendorff's alpha at 0.11 or below in every training setting 2107.00061 §3.2 2107.00061 tab. 2. Untrained raters' free-text reasons cited form most often, and example training moved the mentions toward content 2107.00061 tab. 3. The authors advise against small untrained human evaluations of current models [arxiv:2107.00061 §5]. The GPT-3 snapshot and the GPT-2 decoding settings are not stated 2107.00061 Pitfalls. Card: All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text.

Safety rates scored by a judge model or a fixed rule

For safety tests the usual number is the share of prompts that get an unsafe answer. MM-SafetyBench pairs 1,680 harmful questions across 13 policy scenarios with generated images and scores each answer with a GPT-4 judge 2311.17600 §4.2. Across scenarios, the average attack success for LLaVA-1.5-7B rises from 41.01% with text alone to 71.52% with a typography image 2311.17600 tab. 3. A safety prefix on the same model with both images brings the average down to 17.08%, while a trained harm detector gives 26.11% 2311.17600 tab. 4. Because the judge is a model, check its agreement with people on a sample before quoting the rate. Card: MM-SafetyBench.

The mental-health safety study scores stigma and crisis replies with fixed rules and a human reference. Stigma is a constructed score over 72 vignettes and six scored questions, and "do not know" answers are dropped 2504.18412 §4.1. Crisis replies to ten stimuli are judged by gpt-4o against a yes/no check that depends on each stimulus. The judge was checked on 100 replies, with Fleiss' kappa of 0.96 2504.18412 §5. Sixteen therapists answered half of the stimuli each and reached 93% appropriate replies, against under 80% overall for the gpt-4o and Llama 3.1 models 2504.18412 §5.1. The tested model is also the judge, and some models gave usable answers on far fewer prompts than others 2504.18412 Pitfalls. Card: Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers.

Two system cards show closed-model safety rates with their graders named. The o1 card scores disallowed-content and jailbreak sets with a GPT-4o autograder, and scores its biorisk answers with an o1-preview autograder that was checked against one trusted expert 2412.16720 §4.1.1 2412.16720 §5.5.1. Its chain-of-thought monitor is a GPT-4o judge, validated at 92% accuracy on a small internal set 2412.16720 §4.3.1. The GPT-4o card scores audio by converting text benchmarks to speech with a TTS system and scoring only the text of the reply 2410.21276 §3.2. Its voice output classifier reaches 0.96 precision and 1.0 recall in English 2410.21276 tab. 1. The authors say TTS is lossy for equations, code and formatting-heavy text, so the speech results mix model and TTS errors 2410.21276 §3.2. Cards: OpenAI o1 System Card and GPT-4o System Card.

Ground-truth record checks with reference-free bounds

Here the answer is checked against a structured record where one exists. For nine of fourteen legal tasks, the paper compares a greedy answer with court metadata and does not count abstentions as hallucinations [arxiv:2401.01301 §4.2] 2401.01301 eq. 1. The other five tasks have no reference. For these the paper samples two answers at temperature 1, asks a GPT-4 judge whether they contradict each other, and counts a contradiction as a hallucination. This gives a lower bound only, since two consistent answers can still be wrong 2401.01301 §4.3 2401.01301 eq. 2. On existence questions for Supreme Court cases, zero-shot GPT-4 gives 0.204 and Llama 2 gives 0.303 2401.01301 tab. 2. For the central holding of a Supreme Court case, zero-shot GPT-4 gives 0.600 2401.01301 tab. 4. The paper's headline puts GPT-4 at 58% and Llama 2 at 88% on direct questions 2401.01301 §1. The paper also estimates calibration: each answer's confidence is the share of 10 temperature-1 samples that match the greedy answer, and the pooled expected calibration error is 0.190 for GPT-4 and 0.421 for Llama 2 2401.01301 app. D.1 2401.01301 §5.3. Card: Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.

Fixed scoring rules: exact match, keyword and substring checks

Medical hallucination tests score each answer with a fixed rule, and the rule decides the result. Med-HALT has reasoning tests on multinational exam items, including fake questions and a none-of-the-above option, and memory tests that ask for PubMed titles, PMIDs and links 2307.15343 §3.1 2307.15343 §3.2. An answer is worth +1 if correct and −0.25 if wrong 2307.15343 §5.3. Llama-2 70B leads the false-confidence test at 42.21% accuracy, and Falcon 40B reaches 99.89% on fake questions 2307.15343 tab. 2. Those fake-question numbers need care. The paper states no matching rule for them, and the released code counts an answer as correct when it contains any term from a list of negative phrases 2307.15343 Paper vs code. Treat such accuracy as an upper bound. Two more checks are needed before reading the score column. Its formula divides by a constant rather than by the number of items, so it differs from the paper's average 2307.15343 §5.3 2307.15343 eq. 1. And the paper's open-model setting says sampling, while the released script runs greedy decoding unless a flag is passed 2307.15343 §5.2. Card: Med-HALT.

OCRBench uses the same kind of rule for reading text in images. It has 1,000 questions across five jobs: text recognition, scene-text and document questions, key-information extraction and handwritten maths 2305.07895 §3.3 2305.07895 tab. 5. A reply scores 1 if a ground-truth string appears anywhere in it 2305.07895 §3.1. Final scores out of 1,000 are 659 for Gemini, 645 for GPT4V and 514 for Monkey 2305.07895 tab. 3. The released scorer also lower-cases the text, and removes every space for handwritten maths, which the paper does not state 2305.07895 Paper vs code. Substring matching lets extra words pass, so a negated answer that contains the string also counts 2305.07895 Pitfalls. GPT4V refused 84 images on safety grounds, and the scorer counts each refusal as wrong 2305.07895 §3.3. The per-model token caps differ and are not stated in the paper, so compare models only where the caps match 2305.07895 Pitfalls. Card: OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models.

Bias and faithfulness probes

These tests look for a systematic effect rather than a single error rate. Each needs a control condition.

BBQ builds short question sets by hand, each with an ambiguous and a disambiguated context, and asks models to choose between a target group, a non-target group and an unknown answer 2110.08193 §3. The bias score is twice the aligned share of non-unknown answers minus one. For ambiguous contexts it is also scaled by accuracy, because an unscaled score would give an always-biased model the same extreme value in both context types 2110.08193 §5. UnifiedQA-11B reaches the highest overall accuracy at 77.8%, and RoBERTa-Base the lowest at 61.4% [arxiv:2110.08193 §6]. The authors call a low score no proof of absence of bias, because the scores rest on 25 templates per category 2110.08193 §7 [arxiv:2110.08193 §9]. Card: BBQ.

StereoSet gives each target three completions: a stereotype, an anti-stereotype and an unrelated option. Its context association tests cover gender, profession, race and religion, with 16,995 items over 321 target terms [arxiv:2004.09456 tab. 1]. Two scores are computed per target. The language modeling score (lms) is the share of items where the model prefers a biased option over the unrelated one. The stereotype score (ss) is the share where it prefers the stereotype over the anti-stereotype, and 50 is the ideal [arxiv:2004.09456 §6.2]. The ICAT combines them as lms times the smaller of ss and 100 minus ss, divided by 50 2004.09456 §6.2. The best test ICAT the paper reports is 73.0, for GPT-2 small, against an ideal of 100 2004.09456 tab. 4. The test split is hidden, and the released scoring code differs from the paper in several places, including how ties are counted 2004.09456 Paper vs code 2004.09456 Pitfalls. Card: StereoSet: Measuring stereotypical bias in pretrained language models.

The chain-of-thought faithfulness study uses the same logic with a biased hint. It inserts a suggested answer or a fixed answer position into the prompt and checks whether the model's written reasoning mentions that hint. With the suggested answer in place, GPT-3.5 zero-shot accuracy on 13 BIG-Bench Hard tasks falls from 59.6 to 23.3 2305.04388 tab. 8. For Claude 1.0 the fall is from 65.3 to 34.7 2305.04388 tab. 8. Its BBQ runs ask how often an unfaithful explanation is explained by stereotypes, and a debiasing instruction moves Claude 1.0 zero-shot from 54.5% to 45.4% 2305.04388 tab. 5. The authors say their mention check is a sample: 0 of 234 BBH explanations mention the hint 2305.04388 app. B. Passing the test does not establish faithfulness, and the test only probes small input changes 2305.04388 §6. Card: Language Models Don't Always Say What They Think.

CAMeL tests whether Arabic language models prefer Western entities in an Arab context. It has 628 Arabic prompts, 250 with Arab cultural context and 378 neutral, and 20,368 entities in eight types 2305.14456 §3. The Cultural Bias Score is the share of Arab–Western entity pairs where the model gives the Western entity the higher probability at the mask, averaged over prompts 2305.14456 §4.4. Averaged over five runs, the CBS on the contextualised prompts runs from 45.52 to 63.47 across the models in the table, and the ideal is near zero 2305.14456 tab. 10 2305.14456 §4.4. The paper's ranges for this score differ between its text, figure and table captions, so use the table values 2305.14456 Pitfalls. Card: Having Beer after Prayer? Measuring Cultural Bias in Large Language Models.

The response-bias study turns five survey biases into paired questions: acquiescence, allow/forbid wording, response order, odd/even scales and opinion floating. Each pair is compared with a non-bias edit, such as a key typo or a letter swap, that humans are known to ignore 2311.04076 §2.1. Nine models answer each question 50 times at temperature 1, and each shift is tested against zero with a one-sample t-test 2311.04076 §2.2 [arxiv:2311.04076 §2.3]. The authors find that no model matches the human pattern on every bias, and that the RLHF chat models react more to the typo controls than their base models do 2311.04076 §3.1 2311.04076 §3.2. Card: Do LLMs exhibit human-like response biases? A case study in survey design.

The political-bias study probes 14 language models with 62 statements on a social and an economic axis, then pretrains RoBERTa and GPT-2 on partisan news and Reddit corpora to see whether their positions move 2305.08283 §2.1. Pretraining on left Reddit moves RoBERTa on the social axis from 2.97 to −3.03 2305.08283 §4.1. Downstream, the partisan RoBERTa variants stay within about three points of vanilla RoBERTa on balanced accuracy across the three tasks 2305.08283 §4.2 2305.08283 tab. 3. The paper gives no weights for turning the answers into an axis score, so the same answers can give different positions under another scoring rule 2305.08283 §2.1. Card: From Pretraining Data to Language Models to Downstream Tasks.

Cultural values can also be placed on a survey map. Five GPT models answered ten items from a large international values survey, and each set of answers was projected onto the survey's two-axis map 2311.14096 §4.1 2311.14096 §4.2. The map keeps 107 countries and territories, after five are dropped for missing responses 2311.14096 §4.1. By default the answers sit nearest English-speaking and Protestant European countries, and the farthest are several African-Islamic countries 2311.14096 §2. Cultural prompting, which asks the model to answer as a person born and living in a given country, lowers the mean distance for every model. For GPT-4o it falls from 2.42 to 1.57 2311.14096 §2. The authors say 19 to 29% of countries did not improve or got worse 2311.14096 §3. Each prompt ran once in English at temperature 0, so the only spread measured is across the ten wordings 2311.14096 §4.2. Card: Cultural Bias and Cultural Alignment of Large Language Models.

Simulated participants

A language model can be prompted to play many named participants in a published experiment, and the simulated group can be compared with the human results. The Turing Experiment method asks for a validity rate first. The authors ask for each prompt to be checked on its validity rate before any hypothesis is tested, to avoid p-hacking 2208.10264 §2. The validity rate ranges from 51.0% for the smallest model on the wisdom-of-crowds task to 99.5% for text-davinci-002 on the ultimatum game 2208.10264 tab. 1. Simulated obedience in the Milgram setting is 75 of 100 subjects, against 26 of 40 human subjects in the original study 2208.10264 fig. 7 2208.10264 §6. On the wisdom-of-crowds task the strongest models give estimates that are too accurate, a distortion the authors call hyper-accuracy 2208.10264 fig. 8 2208.10264 app. E. Two cautions apply: the models were probably trained on descriptions of these experiments, and the sampling and repeat counts are not stated 2208.10264 §3 2208.10264 Pitfalls. Card: Using Large Language Models to Simulate Multiple Humans.

Persona prompting can also be scored against a real survey. The cultural alignment study simulates the World Values Survey, wave 7, in Egypt and the US. It uses 30 questions, four paraphrases of each, and 303 matched personas per country 2402.13231 §4.1 2402.13231 §4.2. Each paraphrase is sampled five times at temperature 0.7 and the majority answer is kept 2402.13231 §4.5. Hard alignment is exact agreement, and soft alignment gives partial credit on ordinal items 2402.13231 §4.5 [arxiv:2402.13231 eq. 1]. Averaged over four models, soft alignment is 47.16 for the Egypt survey and 59.07 for the US survey, so the models fit the US survey better 2402.13231 tab. 3. Arabic prompts raise the Egypt score for GPT-3.5 and AceGPT-Chat 2402.13231 tab. 2. The paper's Anthropological Prompting raises GPT-3.5's Egypt soft score from 0.4834 to 0.5102 in its own comparison, but that vanilla row does not match its main tables 2402.13231 tab. 4 2402.13231 Pitfalls. Card: Investigating Cultural Alignment of Large Language Models.

A national survey can serve as the reference for the whole answer distribution, not one label. KorNAT scores Korean social-value items by the share of a 6,174-person survey that chose each option, so a model earns the survey share of its chosen answer 2402.13605 §3.1.3 2402.13605 §4. Averaged over five prompts, PaLM-2 scores 0.331 on SVA, while a constant "Neutral" answer scores 0.190 2402.13605 tab. 1. The paper gives 0.450 as the maximum SVA, calls it empirically calculated, and does not state the formula 2402.13605 §4 2402.13605 Pitfalls. Prompts that ask how someone from Korea would respond raise SVA for every model except Claude-1 2402.13605 tab. 2. Its common-knowledge items come from Korean school materials and are scored by accuracy against a 0.6 reference, a GED pass mark rather than a measured human result. Three of the seven models pass it 2402.13605 §4 2402.13605 tab. 3. The official repository is an evaluation harness, and the leaderboard scorer is not in it, so the metric formulas come from the paper alone 2402.13605 Paper vs code. Card: KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge.

A value survey can also be asked in several languages and wordings, and the answers compared across prompt types. One study asks ChatGPT 24 questions from the Hofstede culture survey in three prompt types: English with a nationality phrase, the target country's language, and English naming the culture setting 2303.17466 §3.1 2303.17466 tab. 1. Consistency is the share of questions on which two prompt types give the same score. English prompts reach 70% or more for every culture except China, and the authors say English prompts make the answers across cultures more alike 2303.17466 §4.1 [arxiv:2303.17466 tab. 3]. Six dimension scores come from a fixed weighted formula, and each is correlated with human reference scores using Spearman's coefficient 2303.17466 §3.1 2303.17466 eq. 1. Only one of the 18 filled cells reaches p < 0.05, and the sample size behind each coefficient is not stated 2303.17466 tab. 5 2303.17466 §4. The paper reports the strongest alignment for American culture, but the Spain cell for prompts 1 and 2 is higher and is not discussed 2303.17466 Pitfalls. The model version, repeat count and decoding are not stated, and language stands in for culture 2303.17466 §4 2303.17466 §6. Card: Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study.

Fit to human reading times

Word surprisal from a language model can also be scored against human reading. The study fits models of four architectures, trained on four corpus sizes, to eye-tracking and self-paced reading data 2006.01912 §2.1. Predictive power is the held-out ΔLogLik gain from adding surprisal to a regression on word length and frequency 2006.01912 §2.2. The authors report that the fit improves as perplexity falls, and that Transformers and the 5-gram model do best at a given perplexity 2006.01912 §3.2 2006.01912 §4. A syntactic generalization score adds nothing once perplexity is controlled [arxiv:2006.01912 §3.3]. The surprisal window differs between the text and the released analysis code, so state which one you use 2006.01912 Paper vs code. Card: Neural surprisal and human reading times.

Repeat runs across serving configurations

Greedy decoding is often assumed to give the same output every time. A study of two reasoning and two non-reasoning 7–8B models runs each model and benchmark in 12 serving configurations: two GPU types, two GPU counts and three batch sizes 2506.09501 §3.1. Under BF16, the greedy accuracy standard deviation across those runs is 9.15% on AIME'24 for DeepSeek-R1-Distill-Qwen-7B. The output length standard deviation is above 9,000 tokens on AIME'24 for both DeepSeek distilled models 2506.09501 tab. 3 2506.09501 tab. 4. The same model under FP32 gives 0 on AIME'24 2506.09501 tab. 3. The paper traces the variation to floating-point rounding and reports it as worst for reasoning models with long outputs 2506.09501 §1 2506.09501 §3.2.

Its LayerCast method stores the weights in BF16 and casts each one to FP32 before its matmul. The paper reports a standard deviation of 0 on AIME'24 across the 12 configurations, the same as FP32 2506.09501 app. F. The released code also changes the attention backend and runs in eager mode, so a LayerCast-versus- BF16 gap cannot be put down to the number format alone [arxiv:2506.09501 Paper vs code]. Random sampling is affected too, but its spread is taken over six configurations, not over repeated runs of one setup 2506.09501 §3.3. Card: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.

Live systems, graded by a checklist and by system state

An agent that must fix a broken system needs a live test, because diagnosis and repair depend on how the system reacts to actions 2605.07161 app. B. SREGym deploys applications on Kubernetes, injects faults and, optionally, background noise, and gives the agent metrics, logs, traces and a cluster command line through MCP servers 2605.07161 §2.2 [arxiv:2605.07161 §2.3]. The agent submits a diagnosis and separately signals that mitigation is finished 2605.07161 §2.1. An LLM judge answers nine yes/no questions in three dimensions, and the pass threshold is 7/9 in the paper [arxiv:2605.07161 §2.5] 2605.07161 app. A. Mitigation is checked against the health of the live system rather than against whether an alert has gone quiet 2605.07161 §2.5. Across 90 problems with three runs each, four agent-model pairs reach 38.1 to 72.6% on diagnosis, 40.4 to 78.5% on mitigation and 26.7 to 60.7% end to end. Diagnosis falls for every pair when noise is on 2605.07161 tab. 3 2605.07161 §3.1. The judge was checked on 100 diagnoses from one domain expert, and the default judge agrees with that expert at Cohen's kappa 0.90 2605.07161 §2.5 2605.07161 tab. 2. The paper's threshold and the released code's threshold differ, 7/9 against 0.70, and the code's dimension weights differ slightly, so state which one a replication uses 2605.07161 Paper vs code. Card: SREGym.

A coding-agent benchmark can grade a whole artifact that only runs in an engine. GameCraft-Bench gives agents 140 Godot game briefs in 15 families [arxiv:2606.17861 §3.1]. The agent submits a project and replayable input traces; a verifier returns a score of zero when the build check fails, replays each trace at 1280×720, samples frames at 2 per second, and asks a multimodal judge to score each rubric item 2606.17861 §3.2 Stage 3 [arxiv:2606.17861 §3.2 Stage 4]. Category scores are weighted 0.15, 0.35, 0.15 and 0.35 by default 2606.17861 §3.2 Stage 5. The best of seven configurations reaches 41.46% overall, and the paper reports no run-to-run variance for agent scores 2606.17861 tab. 4 2606.17861 §4. Ten repeated judge runs on fixed evidence vary by under 0.01 standard deviation on the families checked, but the judge sees only sampled frames and audio is not scored 2606.17861 §5.2 2606.17861 Pitfalls. The human check covers three families and one agent 2606.17861 §5.2. Card: GameCraft-Bench.

Surveys and reviews

Surveys find the literature, but their numbers are secondhand, and some cards give no new measurements. Use them to choose what to read, not as evidence for an effect.

The ChatGPT survey collected 194 arXiv papers that mentioned ChatGPT up to 1 April 2023 and counted them by month and field, and it added no new experiments 2304.01852 §1. Its evaluation section repeats numbers from cited studies, and the survey's judgement is that zero-shot ChatGPT is comparable to fine-tuned BERT-style models but below the state of the art 2304.01852 §3.1. Every task figure comes from another paper, and some are garbled: one summarised result ends mid-sentence 2304.01852 §2.1.6. Card: ChatGPT research survey, April 2023.

The RLHF review is a taxonomy with no new experiments. It cites a human and AI feedback disagreement of up to 10%, 46% and 22% across three cited studies 2307.15217 §4.2.1, and reports that evaluators missed more than half of critical summary errors in one cited study 2307.15217 §3.1.2. The authors call several of their causes unclear 2307.15217 §3.1.1. Card: Open Problems and Fundamental Limitations of RLHF.

A legal-domain review with an evidence map of 327 included technical studies from 2022 to 2025 is another example 2303.09136 §2.1. Its counts describe navigation, not prevalence, and one reviewer screened the titles and abstracts alone 2303.09136 Pitfalls. The card's title and author list do not match the arXiv metadata we hold, so check them before relying on the card 2303.09136 Pitfalls. Card: A Short Survey of Viewing Large Language Models in Legal Aspect.

A three-layer framework for safety evaluation of generative AI (capability, human interaction, systemic impact) maps the 2023 evaluations it could find to six harm areas. It reports few evaluations of human interaction or systemic impact, and few for non-text output 2310.11986 §2 [arxiv:2310.11986 §3.3]. The survey is a snapshot of the authors' own list, not a systematic search, so its counts should not be read as the state of the field 2310.11986 Pitfalls. Card: Sociotechnical Safety Evaluation of Generative AI Systems.

A survey of LLM agent evaluation sorts the work by objective (behaviour, capabilities, reliability, safety) and by process (interaction mode, data, metrics, tooling, contexts) 2507.21504 §2. It names metrics such as pass^k for consistency and LLM and agent judges for scoring [arxiv:2507.21504 §3.3] 2507.21504 §4.3, but few of them are defined, and it reports no measured results 2507.21504 Pitfalls. Card: Evaluation and Benchmarking of LLM Agents: A Survey.

A survey of LLM evaluation organises the field into what to test, where to test and how to test. It catalogues 46 benchmarks in general, downstream and multimodal groups, and lists six human-evaluation criteria [arxiv:2307.03109 §4] 2307.03109 §5.2. It has no experiments of its own, and it concludes that no single protocol or benchmark is shown to be the most useful [arxiv:2307.03109 §6]. Its metric formulas are written in its own notation, so take them from the original metric sources 2307.03109 §5.1. Card: A Survey on Evaluation of Large Language Models.

A narrative review of LLMs in mental health education, assessment and intervention rates each risk for the three uses 2403.14814 tab. 1. It runs no experiments, and the numbers it quotes come from cited studies, some of them preprints 2403.14814 Pitfalls. Card: The Opportunities and Risks of Large Language Models in Mental Health.

A scoping review covers 118 peer-reviewed studies from 2017 to 2022 that use language models to automate educational tasks, sorted into 53 use cases 2303.13379 §4.1 2303.13379 tab. 1. Only 11 of the 118 released both code and data, and 107 cannot be replicated from what they publish 2303.13379 §4.2.3. Its "LLM" includes BERT-family models, which make up 109 of the 118 studies, so the counts say little about GPT-scale models 2303.13379 §4.1 2303.13379 Pitfalls. Study quality was not assessed, and the reported scores use different metrics and datasets, so they should not be ranked against each other 2303.13379 §5.3. Card: Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review.

A broad survey of LLMs with 10 billion or more parameters covers the building blocks, training, tuning, efficiency and evaluation 2307.06435 §1. It runs no experiments. Its one results table lists top scores per benchmark, for example 86.4 for GPT-4 on MMLU with 5 shots, and the survey does not say which numbers were rerun 2307.06435 tab. 13. Shot settings differ between rows, and some tables use undefined symbols, so check the original source before you copy a score or a setting 2307.06435 Pitfalls. Card: A Comprehensive Overview of Large Language Models.

A review of LLM reasoning and autonomous agents groups about 60 benchmarks into a taxonomy and compares agent frameworks and three message protocols, MCP, ACP and A2A 2504.19678 §I 2504.19678 §III. It describes no search protocol and gives no selection rule, so its benchmark list is not a systematic sample 2504.19678 §I. Every figure is quoted from a cited work. One example is a claimed 90% agreement with human judges for an agent-as-a-judge method, against 60 to 70% for LLM judges on 55 tasks 2504.19678 §III. Its Humanity's Last Exam figures disagree with the Humanity's Last Exam card (see Contradictions). Card: From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review.

An opinionated survey of open LLM challenges places evaluation among the problems that slow research: benchmarks that are brittle to prompt and protocol, static human-written references that go stale, and generated-text detectors that paraphrasing can evade 2307.10169 §2.11 2307.10169 §2.12 2307.10169 §2.13. It runs no experiments of its own. Its one detector number is quoted from another paper, where paraphrasing cut DetectGPT accuracy from 70.3% to 4.6% 2307.10169 §2.13. Card: Challenges and Applications of Large Language Models.

A review of bias in LLMs covers about 200 papers from 2016 to 2024 and groups evaluation into data-level, model-level, output-level and human-involved methods 2411.10915 §1 2411.10915 §4. For human-involved checks it asks for an explicit protocol: who the raters are, the guidelines and rubric, a pilot for calibration, a rule fixed in advance for disagreement, and reported agreement statistics 2411.10915 §4.4. Its cost comparison for mitigation methods is the authors' estimate, not a measurement 2411.10915 tab. 5. Card: Bias in Large Language Models: Origin, Evaluation, and Mitigation.

An audit blueprint can stand in for measurement where none exists yet. A three-layer proposal audits a provider's governance, the model before release, and each product built on it 2302.08500 §4.1. Governance audits need white-box access, model audits medium access, and application audits black-box or input access 2302.08500 §4.2 2302.08500 §4.3 [arxiv:2302.08500 §4.4]. The paper gives four minimum model characteristics (performance, robustness, information security, truthfulness) but no pass or fail threshold for any of them 2302.08500 §4.3. It has no experiments; its evidence is a literature review and expert interviews 2302.08500 app. A. Card: Auditing large language models: a three-layered approach.

Generated personas compared with a default group

Instead of a word list, some bias checks ask the model to describe a person from each named group, then find the words that set that group apart from a default group. Marked Personas uses 15 gender-by-race groups, six prompts and 15 samples per group and prompt, on GPT-4 and text-davinci-003 [arxiv:2305.18189 §4]. The unmarked defaults are White and man, and a word counts only if its log-odds difference is significant (z > 1.96) against both defaults 2305.18189 §3.2. The authors report that generated personas use more stereotype-lexicon words than human-written personas for the same prompts, but the comparison appears only in a figure 2305.18189 §5 fig. 1. The top marked words are mostly positive in sentiment, with a mean VADER score of 0.05 and none negative, so a sentiment filter would not catch them [arxiv:2305.18189 §6] tab. 2. The same analysis is run on short stories 2305.18189 §7. The released code strips every non-letter character, so contractions lose their apostrophe, and it uses all texts as the prior, not only the other ones 2305.18189 Pitfalls 2305.18189 Paper vs code. Card: Marked Personas.

Attacks on the instruction, not the input

Prompts are reused across many inputs, so a small edit to the instruction can change every answer. PromptRobust leaves the sample alone and perturbs only the prompt, with seven black-box text attacks at character, word, sentence and semantic level 2306.04528 §1 2306.04528 §2.2. The attacks are run on four prompt types across 13 datasets and eight tasks [arxiv:2306.04528 tab. 1]. The drop is reported as a performance drop rate against the clean prompt on the same sampled set 2306.04528 §3. Averaged over models and prompts, the word-level attacks cost most, at 0.31 for TextFooler and 0.33 for BertAttack, and the sentence-level ones cost least, at 0.12 for CheckList and 0.11 for StressTest 2306.04528 tab. 2. Few-shot prompts held up better than zero-shot prompts on almost every dataset 2306.04528 tab. 4, with WNLI as an exception 2306.04528 Pitfalls. Among the models, Flan-UL2 and GPT-4 average 0.08 and Vicuna-13B 0.69 2306.04528 tab. 3. Three cautions apply. Clean baselines differ widely, and Vicuna's clean SST-2 score is 21.12, which inflates its drop 2306.04528 tab. 9 [arxiv:2306.04528 Pitfalls]. GPT-4 is not attacked directly, and its prompts were transferred from ChatGPT, so its low drop may partly reflect weak transfer 2306.04528 §2.1 2306.04528 §4.1. The 85% acceptance claim for semantic-level prompts is not backed by the unpublished table in the source 2306.04528 Pitfalls. Card: PromptRobust.

Per-skill suite from human negotiation dialogues

A single negotiation score mixes reading, modelling the partner and choosing a move. This framework writes 35 zero-shot tasks from four bilateral negotiation datasets, CRA, DND, CA and JI, and tags each task by time stage, objectivity and task type 2402.13550 §2.2 2402.13550 §2.2.1 2402.13550 §2.2.3. Each task uses at most 200 test instances 2402.13550 §2.1. Two baselines are used: a majority answer and a Flan-T5 base fine-tuned per task 2402.13550 §3.1. GPT-4 is best on most tasks. On end-of-dialogue comprehension it averages 65.3% accuracy [arxiv:2402.13550 fig. 3], and on subjective partner-modelling its correlation with human ratings is the highest, at 0.39, while Flan-T5 reaches 50.5% accuracy [arxiv:2402.13550 §4.3]. Start-stage comprehension is solved almost exactly by a rule-based parser, so those items measure little 2402.13550 §4.1. The majority answer beats GPT-4 on one campsite task, 0.142 against 0.083 2402.13550 app. A. The released code does not compute the PCC or BERTScore reported in the paper, and a failed or over-length GPT call leaves a fixed placeholder string as the stored answer, which is then scored 2402.13550 Paper vs code. Card: Are LLMs Effective Negotiators?.

Live forecasting, scored for calibration and money

Forecasting needs its own scoring, because a sharp but wrong forecast and a vague one can score close together. Prophet Arena draws binary prediction-market events, gives every model the same searched sources and market snapshot, and scores the probabilities after resolution 2510.17638 §2.2 [arxiv:2510.17638 app. A.2]. It uses three metrics: the Brier score, a binned calibration error with ten bins, and the average return of a one-dollar bet [arxiv:2510.17638 §3.1] 2510.17638 app. B.2. On 1,367 events, Brier scores lie between about 0.18 and 0.22, against 0.25 for random guessing, and the best model, GPT-5 with reasoning, scores 0.184 against 0.187 for the market 2510.17638 tab. 2 2510.17638 §3.2.2. Every highlighted model is better calibrated than the market, but none reaches the break-even return of 1, and the best reaches 0.943 2510.17638 §3.2.2. The events are 76% sports, so the results mostly describe that category 2510.17638 §3.2.2. The prompt includes the market price, so the market is a reference point rather than an independent rival 2510.17638 app. E. Forecasts within three hours of resolution are dropped 2510.17638 §3.2.1. Card: LLM-as-a-Prophet.

When to use which

Implementation sketch

Python-flavoured sketch of a run that records what the cards say is usually missing. Off-format handling is left to you, because the papers do not state how they score it.

# Evaluation run with the cautions from the cards
# Input:  dataset D of (x, gold), model M, instruction P, decoding settings,
#         list of formats F (for the spread check)
# Output: report with accuracy per format, parse failures, target log-likelihood

def evaluate(D, M, P, decoding, F):
    # Record snapshot, prompt and decoding; papers often leave these out
    # [arxiv:2302.06476 Limitations] [arxiv:2303.12712 §3.1.1]
    report = {"model": M, "prompt": P, "decoding": decoding, "failures": []}
    for fmt in F:
        # Same items, only the format changes  [arxiv:2310.11324 §3]
        for x, gold in D:
            # Instruction plus input, no demonstrations  [arxiv:2302.06476 §3]
            reply = M.generate(fmt(P, x), **decoding)
            # Parse strictly and log misses instead of guessing
            # [arxiv:2302.06476 §4.2.3] [arxiv:2302.06476 §4.2.8]
            answer = parse_strict(reply, allowed=label_set(gold))
            if answer is None:
                report["failures"].append((fmt, x))
                continue
            report.add_correct(fmt, answer == gold)
            # Log the gold answer's log-likelihood too; accuracy can stay at
            # chance while this still moves [arxiv:2206.07682 app. A.1]
            report.add_loglik(fmt, M.logprob(fmt(P, x), gold))
    # Report the spread over formats, not one format's score
    # [arxiv:2310.11324 §4.2]
    report["spread"] = max(report.acc) - min(report.acc)
    # Re-run a subset with changed numbers to check for memorisation
    # [arxiv:2303.12712 fig. 4.2]
    report["resampled"] = resample_check(M, P, D)
    return report

Contradictions

• Fine-tuned baselines. The ChatGPT study concludes that its zero-shot results underperform earlier fine-tuned methods in most cases [arxiv:2302.06476 §4.3]. The survey judges the same kind of zero-shot results as comparable to fine-tuned BERT-style models while below the state of the art [arxiv:2304.01852 §3.1]. The two statements compare against different baselines, and we do not resolve the difference in framing.
• Same model, different GSM8K numbers. text-davinci-003 scores 59.5 on GSM8K in one study 2302.06476 tab. 2 and 61.3 in another [arxiv:2303.12712 tab. 3]. The cards do not say whether prompts, decoding or sampling differ between the two, so the gap is unexplained.
• Size of prompt sensitivity. On MMLU-Pro, prompt changes move scores by about 2 points, with a maximum of 3.74 2406.01574 §6.3. For few-shot prompts that differ only in format, the median spread across tasks is 7.5 points, and up to 76 points for one model 2310.11324 §4.2 2310.11324 §1. The studies vary different things, prompt wording and answer format, on different tasks and models, so the figures do not directly compare.
• Human agreement with a judge. GPT-4 agrees with human reviewers on 50.8% of pairwise picks 2303.12712 §7.2.2. The Likert study reports story-level Kendall's tau of 0.14 to 0.43 between LLM and teacher averages 2305.01937 tab. 2. The rubric judge on exam answers reports question-level Spearman of 0.825 to 0.864 against teachers 2305.12474 tab. 4. The tasks and measures differ, so these numbers do not directly compare. GEMBA's 89.8% is system-level pairwise accuracy against MQM labels, a third measure 2302.14520 tab. 2.
• Do prompted LLMs match fine-tuned baselines? On relation extraction, in-context GPT-3 is close to the supervised rows, with F1 76.53 on CoNLL04 and 82.66 on ADE 2305.05003 tab. 2, but only after human re-judging of exact-match errors 2305.05003 §3.2. On 12 biomedical tasks, the fine-tuned baselines beat the best prompted LLM on 10, with a macro-average of 0.6536 against 0.4862 2305.16326 p. 6 2305.16326 tab. 3. The tasks and scoring rules differ, so neither result settles the question.
• Do more examples help? For Qwen1.5 models, few-shot examples lower prompt sensitivity, with the largest drop from zero to one shot [arxiv:2410.12405 §3.4]. In the biomedical study, dynamic one-shot is sometimes below static one-shot, and five dynamic shots are not the best GPT-4 setting on 8 of 12 datasets 2305.16326 p. 8. The two studies measure different things, so neither result overrides the other.
• Automatic metrics versus human preference. On news summaries, the human majority prefers GPT-3 while ROUGE and several factuality scores rank it lower 2209.12356 tab. 3 2209.12356 tab. 4 2209.12356 tab. 5. The same paper argues this points to a weakness of the metrics for prompted systems 2209.12356 §4. The cards do not settle which side is right for prompted systems.
• Human baseline on LeetCode, one paper. Medium-level human accuracy is 37.7% in one table and 38.7% in another from the same paper [arxiv:2303.12712 tab. 2] 2303.12712 tab. 9. The overall human figure is not the plain mean of the level figures under either value.
• Humanity's Last Exam size and scores. The benchmark card gives 2,500 questions and accuracy from 2.7% to 13.4% for eight models [arxiv:2501.14249 §4.2] 2501.14249 tab. 1. The review gives 3,000 questions and says accuracy is below 10% for the frontier models it lists 2504.19678 §III. The review does not name its models or dates, so the two cannot be reconciled from the cards.
• Reasoning on analogies versus logic failures. A zero-shot study of text-davinci-003 reports accuracy at or above the average student on letter-string analogies and on three of four verbal-analogy sets 2212.09196 §2.2 2212.09196 §2.3. A failure archive of ChatGPT lists logic errors, such as contradicting itself on planetary densities and failing a Wason selection task 2302.03494 §2.2, but gives no counts. The two use different models and item types, and the archive cannot say how often such errors occur, so the results do not directly disagree.

Papers

• Emergent Abilities of Large Language Models: defines emergence on scaling curves, lists thresholds and checks whether loss and partial-credit metrics show the same jump.
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?: zero-shot run of ChatGPT and GPT-3.5 on 20 datasets, with per-task results and parsing caveats.
• Sparks of Artificial General Intelligence: Early experiments with GPT-4: novel-task probing, memorisation checks, coding and math tests, and a GPT-4 judge against humans.
• ChatGPT research survey, April 2023: bibliometric and narrative survey of early ChatGPT studies, with no new experiments.
• Can Large Language Models Be an Alternative to Human Evaluations?: LLM Likert ratings of story and adversarial text, compared with three teachers per set.
• Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models: reference-based and reference-free hallucination rates on US federal case questions, with calibration.
• MMLU-Pro: harder ten-option benchmark with chain-of-thought results per subject and a prompt-sensitivity check.
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design: spread of accuracy across grammar-generated prompt formats, with a Thompson-sampling search.
• The Belebele Benchmark: reading comprehension in 122 language variants, comparing shots, fine-tuning and translate-test.
• The Illusion of Thinking: complexity sweeps on four puzzles, comparing reasoning and non-thinking models at matched budgets.
• EgoSchema: long-video multiple-choice benchmark with a zero-shot protocol and a human accuracy estimate.
• Humanity's Last Exam: expert-written frontier benchmark with accuracy and RMS calibration error.
• Evaluation of OpenAI o1: case study of o1-preview across 27 tasks, with small samples per task.
• A Survey on LLM-as-a-Judge: taxonomy of judge design and reliability, with a small judge comparison.
• MM-SafetyBench: multimodal attack success rate across 13 scenarios, with a GPT-4 judge.
• Med-HALT: medical hallucination tests with a fixed scoring rule and recall checks.
• BBQ: A Hand-Built Bias Benchmark for Question Answering: paired ambiguous and disambiguated contexts, with a bias score scaled by accuracy.
• Language Models Don't Always Say What They Think: biased-hint test of whether chain-of-thought explanations mention the hint.
• From Pretraining Data to Language Models to Downstream Tasks: political positions of language models, and how partisan pretraining moves them and downstream classifiers.
• Using Large Language Models to Simulate Multiple Humans: Turing Experiments that simulate many named participants and compare with published human results.
• News Summarization and Evaluation in the Era of GPT-3: human best-worst study of prompted summaries, with automatic metrics that disagree with it.
• The Authenticity Gap in Human Evaluation: system-level probability protocol for human preference, compared with averaged Likert ratings.
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks: minimal-edit probes of two vignettes, with probability scoring of completions.
• Open Problems and Fundamental Limitations of RLHF: taxonomy of open problems, with cited figures and no new experiments.
• A Short Survey of Viewing Large Language Models in Legal Aspect: legal-domain evidence map and framework; the card flags a title and author mismatch.
• Neural surprisal and human reading times: surprisal from four model architectures against eye-tracking and self-paced reading, with held-out predictive power.
• Revisiting Relation Extraction in the era of Large Language Models: few-shot GPT-3 with human re-judging of exact-match errors, and a distilled Flan-T5 compared with supervised baselines.
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark: zero-shot scores on national exam questions, with a rubric judge checked against teachers.
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models: CAMeL prompts and a masked-entity Cultural Bias Score for Arabic models.
• Benchmarking large language models for biomedical natural language processing applications and recommendations: 12 biomedical datasets under zero-shot, few-shot, dynamic few-shot and fine-tuning, against published baselines.
• Sociotechnical Safety Evaluation of Generative AI Systems: three-layer framework and a survey of 2023 generative-AI safety evaluations, with no experiments.
• Do LLMs exhibit human-like response biases? A case study in survey design: paired survey-bias questions with non-bias typo controls, across nine models.
• Investigating Cultural Alignment of Large Language Models: persona prompting against the World Values Survey in Egypt and the US, with Anthropological Prompting.
• The Opportunities and Risks of Large Language Models in Mental Health: narrative review of opportunities and risks, with a risk-by-use table and no experiments.
• ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs: per-instance prompt sensitivity score across prompt variants, with a decoding-confidence check.
• Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers: stigma vignettes and scripted crisis stimuli, judged by gpt-4o and compared with therapists.
• Evaluation and Benchmarking of LLM Agents: A Survey: taxonomy of agent evaluation objectives and process, with named metrics and no experiments.
• StereoSet: Measuring stereotypical bias in pretrained language models: crowdsourced context association tests scoring stereotype and language modeling, combined as ICAT.
• LLaMA: Open and Efficient Foundation Language Models: base-model multiple-choice scoring by likelihood, with its normalisation rules and zero-shot tables.
• A Survey on Evaluation of Large Language Models: taxonomy of what, where and how to evaluate LLMs, with a benchmark catalogue and no experiments.
• Revolutionizing Finance with LLMs: An Overview of Applications and Insights: finance survey with a small zero-shot GPT-4 evaluation on six datasets and no baseline.
• GPT-4o System Card: speech-focused safety evaluations, audio benchmarks through TTS, and Preparedness scores.
• OpenAI o1 System Card: refusal, jailbreak, hallucination and bias results with named graders, pre- and post-mitigation Preparedness rows, and multilingual MMLU.
• Benchmarking Large Language Models for News Summarization: human ratings of ten LLMs on two news datasets, with a writer-made reference and pairwise preference.
• ChatGPT: Jack of all trades, master of none: ChatGPT on 25 analytical tasks, scored against the best system per task, with personalised prompts.
• Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review: scoping review of 118 education studies on replicability, transparency and ethics; no experiments.
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference: greedy and sampled variation across GPU and batch settings, with the FP32 control and LayerCast.
• ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports: 15 radiologists' Likert ratings of ChatGPT rewrites of three fictitious radiology reports, for correctness, completeness and harm.
• How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation: three GPT models on 18 translation directions, with shot selection, document windows and a hybrid with a commercial translator.
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality: GEMBA, zero-shot GPT scoring of translations with or without a reference, compared with WMT22 metrics.
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models: 1,000-item OCR benchmark over five tasks, scored by substring match against the ground truth.
• KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge: survey-share scoring of Korean social-value items and accuracy on school-knowledge items, for seven LLMs.
• Using cognitive psychology to understand GPT-3: classic psychology experiments run trial by trial on GPT-3, compared with human data.
• A Comprehensive Overview of Large Language Models: broad LLM survey with copied architecture settings and a top-score table, with no experiments.
• From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review: benchmark taxonomy and agent review with protocol comparison, quoting figures from cited works.
• All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text: untrained and trained crowd raters judging whether GPT-3 and GPT-2 text is human-written.
• Emergent Analogical Reasoning in Large Language Models: zero-shot analogy puzzles on text-davinci-003, scored by log-probability and compared with students.
• Self-Instruct: Aligning Language Models with Self-Generated Instructions: model-written instruction data, a ROUGE-L comparison on unseen tasks and a small expert rating.
• A Categorical Archive of ChatGPT Failures: qualitative archive of ChatGPT failures in eleven categories, with no counts.
• Challenges and Applications of Large Language Models: opinionated survey of open LLM challenges in design, behaviour and science, with no experiments.
• Bias in Large Language Models: Origin, Evaluation, and Mitigation: review of bias origin, evaluation and mitigation, with the estimated cost of each mitigation class.
• Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison: four judge designs for support labels on three QA datasets, with a single-author label check.
• Large Language Models Encode Clinical Knowledge: MultiMedQA benchmark, few-shot, self-consistency and soft-prompt results, and clinician and lay rating of long answers.
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning: zero-shot ChatGPT on seven NLP tasks in 34 to 37 languages, against supervised baselines.
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion: trained pairwise ranker and fuser over 11 open models, with a ChatGPT pairwise judge on MixInstruct.
• Cultural Bias and Cultural Alignment of Large Language Models: five GPT models placed on a cultural survey map, with and without a country-identity prompt.
• SREGym: live Kubernetes fault benchmark, with a checklist judge for diagnosis and system-state checks for repair.
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs: k-shot likelihood probes of GPT-3 on social commonsense and false-belief stories, with factual items as a control.
• Auditing large language models: a three-layered approach: proposed governance, model and application audits, with no experiments.
• Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study: Hofstede survey items in three prompt types, scored with a weighted formula and compared with human scores.
• Leveraging LLMs for Translating and Classifying Mental Health Data: depression-severity classification in English and in LLM-translated Greek, with one model and one prompt.
• GameCraft-Bench: 140 Godot game-building tasks, graded by replay, frame sampling and a rubric judge.
• Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models: generated personas for 15 gender-by-race groups, with marked words against White and man defaults.
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts: seven text attacks on instructions across 13 datasets and six LLMs, scored by performance drop rate.
• Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues: 35 zero-shot negotiation tasks from four dialogue datasets, against majority and fine-tuned baselines.
• LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena: live prediction-market forecasting benchmark scored by Brier score, calibration and simulated return.