---
kind: concept
slug: llm-as-a-judge
title: LLM-as-a-judge
summary: >-
  Using a language model to grade other models' outputs in place of human raters. Human agreement runs from 50.8% (forced choice against a gold statement) to 78% (pairwise preference). A zero-shot GPT-4 score reached 89.8% system-level accuracy on machine translation (author-reported). The result depends on the task and the judge: T0 did not reproduce the teacher ordering that text-davinci-003 did.
tags: [evaluation, llm, bias]
papers: ["2303.12712", "2302.14520", "2305.01937", "2305.11206", "2305.12474", "2410.12405", "2411.15594", "2407.15073", "2507.06134", "2507.21504", "2508.07173", "2509.21842", "2603.18161", "2605.16679", "2208.10264", "2303.15056", "2307.03109", "2310.08419", "2310.13548", "2406.12708", "2306.02561", "2404.16130", "2603.28005", "2408.06292", "2512.04797", "2601.04025", "2605.07161", "2508.02276", "2606.17861", "2505.17613", "2507.14267", "2510.17638", "2604.08455"]
related: [llm-evaluation, human-evaluation, text-classification, dialogue-evaluation, explanation-evaluation, instruction-tuning, rlhf, chain-of-thought-prompting, hallucination-detection, ai-generated-text-detection, agent-based-simulation, prompt-engineering, llm-agent, benchmark-validity, policy-gradient, red-teaming, uncertainty-estimation, neural-machine-translation, causal-discovery, multi-agent-systems, ensemble-learning, reward-modeling, graph-based-rag, fault-diagnosis, tool-use, user-simulation, generative-model-evaluation, calibration, multimodal-language-model]
status: draft
updated_at: 2026-11-01
---

## Overview

Scoring a free-text answer is hard when there is no exact key. Similarity
metrics such as ROUGE, BLEU and BLEURT compare wording rather than meaning, and
human ratings are costly and hard to reproduce [arxiv:2303.12712 §7.2.2]
[arxiv:2305.01937 §1]. The idea is to ask a language model to judge instead
[arxiv:2411.15594 §1].

Most variants share two steps. The judge sees the item to be graded, plus
either a reference answer or the same rating question that human raters saw,
and returns a verdict or a score. The verdict is then compared with human labels
on the same items. The variants differ in what the judge is given and in what
it returns, and the papers test different judges, so their agreement figures
are not directly comparable. Some cards report no agreement figure at all, and
for those the comparison step does not apply.

Coverage is thirty-three paper cards. Sixteen use a judge to grade outputs directly:
a forced choice against a gold statement (GPT-4 on TruthfulQA), Likert ratings
of story text with several judge models, zero-shot labels that follow a human
codebook (ChatGPT on tweets and news), GPT-4 as a preference annotator, a
GPT-4-turbo grader with teacher marking criteria on Chinese exam answers, a
judge that scores open-ended answers against a reference model's answer,
several judges voting on safety labels, GPT-4 judging which of two
comments is more positive in a sycophancy test, and a zero-shot GPT score for
each machine-translated segment, with or without a human reference, a
head-to-head win-rate judge on answers to query-focused questions, and ChatGPT
as the pairwise judge in an ensemble evaluation, and a holistic rubric judge
for three-way support labels checked against a reference
[arxiv:2603.28005 §3.2], a GPT-5 mini judge for correctness and errors of
simulated tutoring student turns [arxiv:2601.04025 §2.2], and a multimodal
judge that scores replayed game videos against a hidden rubric
[arxiv:2606.17861 §3.2], a vision-language judge that answers one question per criterion on generated images, audio and interleaved outputs [arxiv:2505.17613 §3.2], and an LLM judge that rates the reasoning behind forecasts on a 1-to-5 scale [arxiv:2510.17638 app. C.6]. Fifteen use a judge
inside a larger pipeline: a survey with a judge experiment across ten
models, a panel of five judges that scores generated research plans in a
virtual-cell agent pipeline [arxiv:2508.02276 § Blinded planning evaluation], a simulation that classifies simulated subjects' actions, an empirical
study that codes a corpus of reviews and essays, agent-safety runs judged
alongside a rule check, a judge used as a reinforcement-learning reward, a
rubric judge in a healthcare agent benchmark, a checklist judge that grades agent diagnoses in a live Kubernetes benchmark [arxiv:2605.07161 §2.5], a judge that scores each attack
attempt in a jailbreak search, a peer-review simulation whose GPT-4 step codes
reasons for acceptance and rejection, a debate in which a judge picks
between two causal-graph proposals, a reward model that rates game-play videos
for self-improvement [arxiv:2512.04797 §4.5], and a reviewer model that scores generated
research papers on a conference review form [arxiv:2408.06292 §4], a judge that
checks each tool parameter before a scientific tool runs, plus a report-time
judge [arxiv:2507.14267 Methods: Safety guard], and a rubric judge whose score is
mixed with rule checks in a mobile-agent benchmark [arxiv:2604.08455 §3.4]. A survey with no experiments also lists
judges among its metric options [arxiv:2507.21504 §4.3]. One more card, a
general survey of LLM evaluation, splits methods into automatic metrics and
human evaluation and describes no model-based judge, so it adds no judge
evidence [arxiv:2307.03109 §5]. The "When to use which" advice is limited to
what these cards report.

## Variants

### Forced-choice pick against a gold reference

The judge gets a gold statement and two options, one of which is the model's
own answer and the other a baseline answer. It writes a similarity and
difference note for each option, then names one option as `BEST` and gives a
short justification. It must pick exactly one [arxiv:2303.12712 §7.2.2]
[fig. 7.5]. On the TruthfulQA pairs, the judge picked the model's own answer
87.76% of the time, the baseline 11.01%, and neither 1.23% [arxiv:2303.12712
tab. 6]. Card: [Sparks of Artificial General Intelligence: Early experiments with GPT-4](/papers/2303.12712).

### Likert rating with no reference answer

The judge gets the same instruction, text and question that human raters got,
and writes a free-text answer. A parser takes the first number in that answer as
the score on a 1-to-5 scale [arxiv:2305.01937 §2.2] [arxiv:2305.01937 app. D.2.1].
For story generation, each of four attributes (grammaticality, cohesiveness,
likability, relevance) gets its own query [arxiv:2305.01937 §3.2]. Against
three certified English teachers, text-davinci-003 and ChatGPT rated
human-written stories above GPT-2 stories on all four attributes, as the
teachers did. T0 and text-curie-001 showed no clear preference
[arxiv:2305.01937 tab. 1] [arxiv:2305.01937 §3.3]. The teacher-versus-LLM
agreement at story level (Kendall's tau) is positive on all four attributes, from
0.14 to 0.43 [arxiv:2305.01937 tab. 2]. Card: [Can Large Language Models Be an Alternative to Human Evaluations?](/papers/2305.01937).

### Zero-shot segment score, averaged per system

The judge gets one source segment and one candidate translation. The human
reference is added only in reference mode. A fixed zero-shot template asks for
a score from 0 to 100, where 0 means no meaning preserved and 100 means perfect
meaning and grammar. Each segment is a separate request [arxiv:2302.14520 §2]
[§2.1] [app. A]. A reply counts only if it is a valid integer in range. Invalid
replies are re-prompted with added randomness [arxiv:2302.14520 tab. 6], and the
paper does not give the step size. Some segments stay unscored after retries
[arxiv:2302.14520 app. B]. The system score is the average of the segment
scores [arxiv:2302.14520 §2.2].

With the reference, GPT-4 reached 89.8% system-level pairwise accuracy on WMT22
MQM, against 85.0% for the non-public MetricX XXL. Without the reference it
reached 87.6%, against 78.8% for COMETKiwi without a reference [arxiv:2302.14520
tab. 2]. The test covers three language pairs and system-level scores only
[arxiv:2302.14520 Limitations]. With the reference, GPT-4's segment-level
Kendall tau-b was 0.36 to 0.38 across en-de, en-ru and zh-en, against 0.37,
0.42 and 0.43 for the best metric in the paper's table. The paper calls its
segment-level performance weaker than the best metrics [arxiv:2302.14520 tab.
4] [§4.4]. Model size matters: with the reference, Davinci-003 reached 88.0%
and GPT-3.5-turbo 86.5%, while Babbage and Curie reached 39.1% and 54.4%
[arxiv:2302.14520 tab. 3]. The paper calls Babbage and Curie close to random,
and says GPT-2 and Ada gave no usable scores [arxiv:2302.14520 §4.3]. Stars
with the reference and Classes without it both reached 91.2% with GPT-4
[arxiv:2302.14520 tab. 3]. The paper names the DA prompt as its main style and does not explain why
[arxiv:2302.14520 §4]. The authors cannot rule out that the test data was in
the model provider's training data [arxiv:2302.14520 Limitations]. Card:
[Large Language Models Are State-of-the-Art Evaluators of Translation Quality](/papers/2302.14520).

### Zero-shot labels from the human codebook

The model gets the codebook that human coders used, with no examples, and
labels one tweet or article per fresh chat. Its labels are scored against
trained research assistants, using only texts on which the two assistants
agreed [arxiv:2303.15056 §4.5] [arxiv:2303.15056 §4.6]. Accuracy on the
content-moderation relevance task was 70% for tweets, 81% for news, 83% for
Congress tweets and 59% for 2023 tweets [arxiv:2303.15056 §2] [fig. 1]. The
authors report accuracy about 25 points above MTurk crowd-workers on average
[arxiv:2303.15056 §2]. Intercoder agreement, the share of texts where two
responses match, was 91% at temperature 1 and 97% at 0.2, against 79% for
trained annotators and 56% for MTurk [arxiv:2303.15056 §2]. This is raw percent
agreement with no chance correction [arxiv:2303.15056 §4.6]. Per-task accuracy
for the other tasks appears only in a plot [fig. 1]. Card: [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](/papers/2303.15056).

### Pairwise preference with a tie option, GPT-4 as annotator

Crowd workers picked the better of two responses or a tie for each test prompt.
The same comparisons were repeated with GPT-4 as the annotator, using the same
instructions [arxiv:2305.11206 §4.1]. GPT-4 agreed with the crowd workers on
78% of comparisons and with the authors on 79%. The paper reports two crowd
workers agreeing on 82% (tie-discounted accuracy) and the authors on 78%
[arxiv:2305.11206 §4.1]. The card does not confirm that the GPT-4 rows use the
same metric as the human rows. Card: [LIMA: Less Is More for Alignment](/papers/2305.11206).

### Graded 1-to-6 helpfulness score with repeated samples

For ablation models, ChatGPT (GPT-3.5 Turbo) graded five sampled responses per
test prompt on a 1-to-6 helpfulness scale. Results are reported with 95%
two-sided confidence intervals [arxiv:2305.11206 §5] [arxiv:2305.11206 app. D].
The paper gives one gap in the text, 0.5 points in favour of filtered Stack
Exchange data, and shows the rest only in figures [arxiv:2305.11206 §5]. Card:
[LIMA: Less Is More for Alignment](/papers/2305.11206).

### Multi-round and position-swapped judging

The judge is run five times on each item, and the label is taken by majority
vote, by mean score or by best score. The paper runs these at temperature 0
[arxiv:2411.15594 §4.4]. Each pair is also judged in both orders, and the
metric is the share of pairs where the two verdicts name the same winner
[arxiv:2411.15594 §4.4]. On GPT-3.5-turbo, majority over five rounds scored
54.68% against a human-label alignment of 54.72% for a single judge, and 70.11%
against 68.78% on position consistency [arxiv:2411.15594 §4.4]. Five rounds at
temperature 0 are not guaranteed to differ, and the paper does not say how they
vary [arxiv:2411.15594 §4.4]. Card: [A Survey on LLM-as-a-Judge](/papers/2411.15594).

### Judge as a classifier inside a simulation loop

In a Milgram-style Turing Experiment, the model writes the simulated subject's
next sentence freely. Two LM judges then classify that sentence. One asks
whether the subject stops, the other whether the subject does not shock, and
each is compared with 0.5 [arxiv:2208.10264 app. F.1]. Across the simulated
subjects, 75 of 100 stayed obedient to the end, against 26 of 40 in the
reference human study [arxiv:2208.10264 fig. 7]. The card gives no agreement
figure for the judges against human labels (not stated). The paper defines the
judge probability as normalized over the valid completions [arxiv:2208.10264 §2],
and the released Milgram code compares the unnormalized value with 0.5; see the
Paper vs code notes on the card [arxiv:2208.10264 §2]. Card: [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](/papers/2208.10264).

### Correctness and error judge in a simulator's evaluation, also a reward

In a study of simulated tutoring students, GPT-5 mini judges each simulated
turn against the real turn as reference, at low reasoning effort. It returns a
correctness score, and an error score for real turns that were incorrect
[arxiv:2601.04025 §2.2] [code: sim_student/eval.py#L83]. The DPO reward is the
average of seven metrics, and this judge is one of the inputs [arxiv:2601.04025
§2.3]. On a subset of 190 turns across 38 dialogues, rated by three evaluators,
the paper's kappa between humans and the metric was 0.6891 for correctness and
0.6127 for errors. Between two evaluators on 20 shared turns it was 0.7187 and
0.6154 [arxiv:2601.04025 tab. 3] [app. E]. The authors warn that training on
LLM-judged metrics can overfit the judge [arxiv:2601.04025 §2.3]. Card:
[Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025).

### Judge as a coder of a text corpus

An LLM judge (gpt-4o) first extracts strength and weakness categories from
ICLR 2026 reviews, then labels each review with them. Category frequencies for
LLM-labelled and human reviews are compared with two-proportion z-tests
[arxiv:2603.18161 §3.1] [arxiv:2603.18161 tab. 1]. The same kind of judge codes
stance and argument types in a 100-person user study [arxiv:2603.18161 §3.2].
Clarity was cited as a strength 32% less often in LLM-labelled reviews than in
human ones, relative to the LLM figure, and the mean score was 4.43 against 4.13
[arxiv:2603.18161 §4.8]. The paper does not report the judge's agreement with
human coders. Its temperature is 1 for the ICLR judge and not stated for the
user study [arxiv:2603.18161 §3.2]. Card: [How LLMs Distort Our Written Language](/papers/2603.18161).

AgentReview uses GPT-4 in the same role, to assign reason categories for
acceptance and rejection to the simulated reviews [arxiv:2406.12708 app. A.1]
[fig. 4]. The card gives no agreement figure for these categories with human
coding (not stated). The judge is one step in a larger simulation, so the
simulation's own reviewer agents are covered on the agent-based simulation
page. Card: [AgentReview: Exploring Peer Review Dynamics with LLM Agents](/papers/2406.12708).

### Teacher marking criteria for subjective exam answers

Teachers wrote marking criteria that split each answer into scoring points. The
judge, GPT-4-turbo at temperature 0, gets the question, the standard answer, the
points, the criteria and the model's answer, and writes a grading process and a
final score [arxiv:2305.12474 §3.4]. Question-level Spearman correlation with
the human scores runs from 0.825 to 0.864 across five models, and Kendall's tau
from 0.685 to 0.740 [arxiv:2305.12474 tab. 4]. With the criteria, GPT-4-0613's
Spearman was 0.854; without them, 0.820 [arxiv:2305.12474 tab. 6]. Converted
totals differ from the human totals by under 2% for sciences and about 5% for
humanities [arxiv:2305.12474 §4.4]. The paper calls matching an answer to its
scoring points a significant challenge for humanities subjects [arxiv:2305.12474
§4.4]. The criteria files are not in the released repository, so the paper's
with-criteria numbers cannot be reproduced from it [arxiv:2305.12474 tab. 6].
Card: [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](/papers/2305.12474).

### Judge score as the input to a sensitivity measure

Here the judge does not measure the model under test directly. Each open-ended
response is scored against the benchmark's reference answer by GPT-4-1106-preview.
AlpacaEval gives a score in [0, 1]. Arena Hard Auto labels map to 0, 0.25, 0.5,
0.75 or 1.0, and each pair is judged in both orders, which gives 1,000 scores
[arxiv:2410.12405 §4.1.1]. The sensitivity measure is the average, over items,
of the mean absolute score gap between the three rewritten prompts of one item
[arxiv:2410.12405 §4.2]. For the five tested models the values are 0.013 to
0.036 on AlpacaEval and 0.249 to 0.266 on Arena Hard Auto. The paper's reference
row, a gap between two models' answers, is 0.167 and 0.275 [arxiv:2410.12405
tab. 2]. The table caption says the two benchmarks cannot be compared directly
[arxiv:2410.12405 tab. 2]. The judge's own run-to-run variance is not measured
[arxiv:2410.12405 §4.1.1]. Card: [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](/papers/2410.12405).

### Several judges voting on safety labels, with manual review of splits

For each response, five judge LLMs vote on three binary labels: whether the
model understood the input, whether the answer is harmful, and whether it
refuses. A 3-2 split goes to manual review [arxiv:2508.07173 §2.3.1]. Only
responses labelled as understood enter the rates, so a model that misreads the
request does not look safe by accident [arxiv:2508.07173 §2.3.1] [eq. 1–2].
Pairwise agreement between the judges was 90.4% on comprehension, 87.5% on harm
and 89.5% on refusal [arxiv:2508.07173 tab. 10]. The table of human accuracy on
500 sampled responses gives 94.2%, 89.6% and 92.8% for the same three labels
[arxiv:2508.07173 §3.3] [tab. 12]. The released repository uses one judge
instead of five and does not route splits to review, so the voting design is
not reproducible from it (see the card's Paper vs code section). Card:
[Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models](/papers/2508.07173).

### Panel of judges from separate model families scoring plans

Here the judged items are research plans, not model answers. In CellForge,
five LLM judges, each from a different model family, score each plan from 1 to
10 on scientific validity, technical feasibility, innovation, experimental
design and impact [arxiv:2508.02276 § Blinded planning evaluation]. The paper
reports off-diagonal inter-judge Pearson correlations of 0.88 to 0.93
[arxiv:2508.02276 fig. 5]. These correlations measure agreement between the
judges, not with people. Three single-cell biology experts gave separate blinded
scores, which the paper says correspond strongly with the judges' scores. It
gives no figure for that correspondence (not stated) [arxiv:2508.02276
§ Discussion]. The same kind of LLM judgement also scores each plan's
confidence inside the agents' discussion, and the paper says this score is
neither a calibrated probability nor a performance measure [arxiv:2508.02276
§ Critic-mediated expert panel]. Card: [AI Scientists for Building Virtual-Cell Models](/papers/2508.02276).

### Judge as the reward in reinforcement learning

The judge can sit inside training as well as evaluation. A trajectory-level
verifier checks that the travel itinerary is feasible, and a turn-level verifier
checks that each tool response is reflected in the plan. Both are LLM judges
(DeepSeek-R1, prompted with human-written rubrics), and the reward is 1 only if
every check passes [arxiv:2509.21842 §3.2.3]. Queries whose sampled
trajectories all earned zero are replayed later [arxiv:2509.21842 §3.3.2]. The
same reward system is also used to score the full evaluation sample. A hand check
of 200 samples found 82% agreement with humans, and the authors estimate 89%
judge accuracy after allowing for an assumed 7% human error rate
[arxiv:2509.21842 tab. 3] [§5.2.1]. Card: [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](/papers/2509.21842).

### Rubric score from a reward model, with a success threshold, for self-improvement

In SIMA 2, a Gemini reward model rates each video of an agent playing a 3D
game from 0 to 100. The rubric covers task completion and directedness, and a
score of 50 or more counts as success [arxiv:2512.04797 §4.5]. The rubric was
calibrated against human preference pairs on a small set, and the paper gives
no size for that set [arxiv:2512.04797 §4.5]. The scored trajectories feed a
loop in which Gemini also proposes the next tasks [arxiv:2512.04797 §4.5]
[fig. 16a]. The paper does not say how scored data becomes a training update,
or how many rounds ran [arxiv:2512.04797 §4.5]. The rubric text is not
published, so the reward cannot be rebuilt from the paper alone. The success
count uses the same reward score. The author-reported result is that the share
of ASKA tasks scoring 50 or more went from under a quarter for the initial agent
to all tasks after self-improvement, with an average above the human reference
score [arxiv:2512.04797 §4.5.1] [fig. 15]. Card: [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](/papers/2512.04797).

### Multimodal judge on replayed game videos, one call per demo

GameCraft-Bench replays each submitted input trace in a Godot game, records the
screen, and samples frames at 2 per second over at most 20 seconds. A
multimodal judge gets the sampled frames and a list of requirements, and returns
strict JSON with one score from 0 to 1 per requirement [arxiv:2606.17861 §3.2]
[app. A]. The default judge is GPT-5.5, and the paper reports its results with
that judge [arxiv:2606.17861 §4]. Ten repeated GPT-5.5 runs on fixed evidence
gave standard deviations of 0.0036 to 0.0050 across two agent configurations
and two game families [arxiv:2606.17861 §5.2] [fig. 9]. The judge's sampling
settings are not stated [arxiv:2606.17861 app. A]. The paper's human check is
labelled a calibration check, not an agreement study. On three game families
for one agent, the human overall score was 22.69 against the judge's 26.02
[arxiv:2606.17861 §5.2] [tab. 5]. The card's repository check found that the
default OpenAI backend sends sampled frames only, and the Gemini backend sends
the whole video. Each backend therefore gives the judge a different input
[code: gamecraft_bench/verifier/judges/openai_gpt.py#L1-L11] [code:
gamecraft_bench/verifier/judges/gemini.py#L1-L5]. A demo whose judge calls all
fail scores zero rather than missing [code: gamecraft_bench/verifier/score.py#L220-L225].
Card: [GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?](/papers/2606.17861).

### Rubric-by-rubric judge with votes, inside a binary reward

A trial passes only if a deterministic check of the persisted record and a
rubric judge both pass [arxiv:2605.16679 §3.2.3]. The judge checks each rubric
with three votes and a strict majority, and a tie counts as a fail
[arxiv:2605.16679 app. F.3]. The judge is Claude Opus 4.7. Several evaluated
configurations are also Claude models, and the paper says this may bias the
results. A cross-family re-grade was planned but not run [arxiv:2605.16679
app. F.3] [app. F.7]. The card's review of the repository found that its default
is one vote unless an environment variable is set, so three-vote runs need that
setting (see the card's Paper vs code section). Card: [CHI-Bench](/papers/2605.16679), a healthcare-workflow agent benchmark.

### Yes/no checklist against a known answer, with a pass threshold

The judge gets the true root cause, the agent's diagnosis and a checklist of
nine yes/no questions in three dimensions. Each dimension has weight one third,
and a diagnosis passes when the weighted score reaches 7 of 9 [arxiv:2605.07161
§2.5] [app. A]. The released code uses 0.70 as its threshold [code:
sregym/conductor/oracles/llm_as_a_judge/rca_checklists.yaml#L3]; see the card's
Paper vs code section. A domain expert labelled 100 diagnoses. Claude
Sonnet-4.6, the default judge, reached Cohen's kappa 0.90 against the expert.
GPT-5.4 reached 0.70 and Kimi K2.5 0.76 against the same labels
[arxiv:2605.07161 §2.5] [tab. 2]. Card: [SREGym](/papers/2605.07161), a live Kubernetes benchmark for SRE agents.

### Pairwise comparison of feedback against a baseline

The judge gets two comments on the same passage: the baseline feedback, given
with no stated opinion, and the feedback given after the user said they liked
or disliked the text. It says which comment is more positive. The paper uses
GPT-4 for this [arxiv:2310.13548 §3.1] [app. A.3]. Feedback positivity is the
share of passages where the opinion-laden comment wins, and the sycophancy
measure is the gap between the positive and negative opinion variants
[arxiv:2310.13548 §3.1]. The figures appear only in a plot, and the text calls
the effect frequent [fig. 1]. The paper gives no judge accuracy for this step,
and the grader temperature is not stated for it [arxiv:2310.13548 app. A.3].
Card: [Towards Understanding Sycophancy in Language Models](/papers/2310.13548).

### Template grader and a yes/no check for admitted mistakes

For free-form answers, GPT-4 at temperature 0 grades each reply with a LangChain
grading template [arxiv:2310.13548 app. A.1]. In the flip-flop test, a second
model reads the reply to "Are you sure?" and decides whether it admits a
mistake. The card records the printed model id as `gpt-3.5-turbo-061`, which
may be a typo, and gives no accuracy for this judge [arxiv:2310.13548 app.
A.4]. The card reports that Claude 1.3 admitted a mistake on 98% of questions
and GPT-4 on 42%, and that the answer changed on 86% and 32% [arxiv:2310.13548
§3.2] [app. A.4] [fig. 2]. The notebook's answer scorer does not implement the
intended test, so check the scorer on sample outputs before using any number
(see the card's Paper vs code section). Card: [Towards Understanding Sycophancy in Language Models](/papers/2310.13548).

### Binary jailbreak judge scoring each attack attempt

In PAIR, an attacker model proposes a prompt, the target answers, and a judge
returns 1 when the pair is a jailbreak and 0 otherwise [arxiv:2310.08419 §3.4]
[alg. 1]. The experiments use Llama Guard with the JailbreakBench prompt
[arxiv:2310.08419 §4 Evaluation] [app. E]. The judge was chosen on 100 prompts
and responses, about half of them jailbreaks, with a majority of three
annotators as the baseline [arxiv:2310.08419 §3.4] [tab. 1]. Agreement with that
baseline was 88% for GPT-4-0613, 81% for the TDC Llama-13B classifier, 80% for
the GCG rule-based judge, 76% for Llama Guard, 74% for GPT-4-Turbo and 66% for a
BERT classifier [tab. 1]. The false-negative rate runs from 7% (GPT-4-0613) to
74% (BERT), and Llama Guard misses 47% [tab. 1]. The judge drives the headline
rates: Llama-2 and the Claude models had the lowest rates in the paper's runs,
and GPT-3.5 and GPT-4 about 50% [arxiv:2310.08419 tab. 2]. The repository's
default judge is rule-based, not Llama Guard (see the card's Paper vs code
section). Card: [Jailbreaking Black Box Large Language Models in Twenty Queries](/papers/2310.08419).

### Debate between proposals, with a judge picking the winner

Here the judge does not grade one output against a rubric. In MAC, an
Affirmative agent proposes a graph or an algorithm. A Negative agent proposes a
plausible alternative from the same evidence. A Judge scores the disagreements
against criteria drawn from the evidence and returns the winner [arxiv:2407.15073
§3.3] [alg. 1] [app. A.1]. The Judge's metric is described only as "a
quantifiable metric", and its formula is not stated [arxiv:2407.15073 app. A.1].
The agent prompts appear only in figures, so the wording must be written from
scratch [arxiv:2407.15073 app. A.1] [app. A.2]. The backbones were Gemini-2.0-Flash,
DeepSeek-R1 and GPT-4o [arxiv:2407.15073 §4.1]. The card records no judge-level
agreement with human labels (not stated).

The full structure was compared with a single-agent variant that has one agent
and no Negative agent or Judge [arxiv:2407.15073 §4.4] [tab. 6]. On Survey with
GPT-4o, F1 rose from 0.36 to 0.79. On Auto with Gemini, it rose from 0.06 to
0.61. Earthquake and Cancer were comparable [arxiv:2407.15073 §4.4]. This
comparison removes the Negative agent and the Judge together, so it does not
isolate the Judge. The full system used 16 to 25 times the tokens of standard
baselines. The single-agent variant was competitive on most datasets at about
half the cost [arxiv:2407.15073 §4.4] [fig. 5]. Card: [Multi-Agent Causal Discovery Using Large Language Models](/papers/2407.15073).

### Head-to-head win rates on criteria the authors wrote

For query-focused summaries, the judge sees two answers to the same question
and compares them on comprehensiveness, diversity and empowerment. Directness
is a control criterion [arxiv:2404.16130 §3.3]. The winner scores 100, the
loser 0, and a tie 50 per condition. Each comparison is judged five times
[arxiv:2404.16130 app. G, tab. 6] [§5.2]. Against vector RAG, the
author-reported win rates for comprehensiveness were 72% to 83% on the podcast
corpus and 72% to 80% on news. Diversity was 75% to 82% and 62% to 71%
[arxiv:2404.16130 §5.1] [fig. 2]. Empowerment was 43% to 58% on podcast and 43%
to 53% on news, and directness favours vector RAG [fig. 2]. The introduction
names GPT-4 as the model, but the exact model used for answering and for
judging is not stated [arxiv:2404.16130 §1] [§4.1.3]. The judge's verdicts were
checked against a claim-based measure. The two agree on 78% of non-tied
comprehensiveness comparisons and on 69% to 70% of diversity comparisons, with
ties at 33% and 39% of pairs [arxiv:2404.16130 §5.2]. Card: [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](/papers/2404.16130).

### Trained pairwise ranker, with ChatGPT judging only the evaluation

LLM-Blender splits the job in two. PairRanker is a trained model, not a
prompted judge. It reads the question with two candidate answers and scores
which one is better, and the pair scores give the ranking of all candidates
[arxiv:2306.02561 §3.2]. Its labels come from BARTScore, not from GPT
[arxiv:2306.02561 §5.1] [app. E]. The LLM judge appears only in evaluation.
ChatGPT compares every candidate pair, and a candidate's GPT-Rank is its number
of wins [arxiv:2306.02561 §5.1] [app. E]. On the MixInstruct test split, with 11
open LLMs as candidates, the fused output reached GPT-Rank 3.01. The best single
model reached 3.90 [arxiv:2306.02561 tab. 2] [§2.2]. The paper does not state
the ChatGPT version, and the ranker agrees with itself under swapping more than
90% of the time [arxiv:2306.02561 app. C.5]. Card: [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](/papers/2306.02561).

### Holistic rubric against claim-splitting, for reference-grounded labels

The judge gets a question, a reference and a candidate answer, and returns a
three-way support label as the first key of a JSON object. The holistic design
works through a four-point checklist: correctness, completeness, unsupported
detail and style robustness. It is told not to split the answer into claims
[arxiv:2603.28005 §3.2]. Two claim-splitting designs are compared with it. One
splits the candidate and checks each claim against the reference (C). The
other splits the reference and checks whether the candidate covers each claim
(R). A third design does both (B) [arxiv:2603.28005 §3.2] [app. G]. On ASQA,
the holistic design was 7.25 to 15.5 points more accurate than C for all four
judges, and used 65 to 81 fewer tokens per call [arxiv:2603.28005 §4.1]
[tab. 3]. R was the most accurate ASQA design for each judge it was run with,
at 28 to 29% more tokens than the holistic design [arxiv:2603.28005 §4.2]
[tab. 2]. On QAMPARI the holistic design was 1.75 to 7.0 points ahead of C and
cheaper, and on TruthfulQA C was at least as accurate as the holistic design
[arxiv:2603.28005 §4.1]. The labels are built from the datasets by rule, and a
separate check on 60 rows by one author is reported apart from them. The paper
says the scores measure agreement with a strict completeness standard, not
factual accuracy [arxiv:2603.28005 §3.1] [§6.1] [§8]. The
ranking also changes with the labels used; see Contradictions. Card:
[Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison](/papers/2603.28005).

### Reviewer model with ensemble, reflection and a score threshold

A reviewer model reads the full text of a generated paper and returns one
review. Five reviews are sampled in parallel, merged into a meta-review, and
numeric fields are averaged over the valid reviews and rounded. The merged
review then goes through five reflection rounds. The reviewer is GPT-4o, given
the NeurIPS review form and one few-shot example [arxiv:2408.06292 §4]
[tab. 6]. Scores are compared with a threshold of 6 or 8 for the calibrated
rows [arxiv:2408.06292 tab. 1]. On 500 ICLR 2022 papers, the calibrated
one-shot GPT-4o row at threshold 6 reached balanced accuracy 0.65 ± 0.04,
against 0.60 ± 0.03 for the uncalibrated one-shot row [arxiv:2408.06292 tab. 1].
The human row, 0.66, comes from a NeurIPS 2021 consistency experiment, not from
the ICLR data [arxiv:2408.06292 tab. 1]. The reviewer's average score correlated
0.18 with the average of human reviewers, and two random human reviewers
correlated 0.14 [arxiv:2408.06292 §4] [fig. 2]. The paper gives an accuracy of
70% for the chosen reviewer, which matches the uncalibrated one-shot row, not
the calibrated one [arxiv:2408.06292 §4] [tab. 1]. The authors say their
generated papers have many failure modes, and that output should be read as
ideas to check, not as established results [arxiv:2408.06292 §5] [§8]. Card:
[The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](/papers/2408.06292).

### Per-criterion questions to a vision-language model, with program checks

For image, audio and interleaved outputs, each check is split into criteria. A
vision-language model answers one question per criterion. Format rubrics ask
for reasoning before a yes/no answer, or a multiple-choice answer for counting
and spatial questions. Content rubrics list the required objects, attributes
and relations, plus explicit negative conditions [arxiv:2505.17613 §3.2] [app.
C.3]. Other tasks use a program or an audio model instead, such as a pitch
tracker, a speech recogniser, or a similarity score against reference clips
[arxiv:2505.17613 §3.2] [app. C.3]. The authors say human-written rubrics
produce fewer hallucinated and near-miss acceptances than automatically
generated question sets such as TIFA [arxiv:2505.17613 §3.2]. Human agreement
was 94.4% overall, 93.5% on images and 91.0% on interleaved audio-text, but the
human check covers 41 of the tasks only [arxiv:2505.17613 §1] [§4.1]. The
thresholds for the similarity checks are described as task-specific, and their
values are not stated in the paper; the released code uses fixed values per
class [arxiv:2505.17613 §3.2] [app. C.1]. Card: [MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613).

### Per-parameter guard before a tool call, with a report-time audit

In DREAMS, a guard runs before each scientific tool call. A deterministic step
first checks that every referenced value exists in the artifact registry and
matches the recorded value. A judge then checks each parameter against the
reason the agent gave for it, and a cross-parameter judge runs only on tools
that need one [arxiv:2507.14267 Methods: Safety guard] [fig. 2]. A warning
verdict still registers the value, with the concern attached, and the guard
fails open on infrastructure errors [arxiv:2507.14267 Methods: Safety guard].
After a report is written, a separate report judge checks each claim against
its cited artifact and walks the provenance graph. Its verdict decides
acceptance, and a call-time verdict can differ from it [arxiv:2507.14267
Methods: Safety guard] [fig. 5]. The judge is Claude Opus 4.8. On a
145-scenario adversarial suite, the five Anthropic models tested made 4 to 17
errors in total, and the production judge made 2 false positives and 5 false
negatives [arxiv:2507.14267 Results: Judge models] [fig. 8]. In the CO/Pt(111)
comparison, the guarded run used about 13 times the input tokens of the
unguarded run, and 100% of its essential steps succeeded against 81%
[arxiv:2507.14267 tab. 6]. Card: [DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation](/papers/2507.14267).

### Reasoning-quality rating of forecasts, checked on human ratings

Prophet Arena scores each forecast's probabilities against outcomes, and it
also asks an LLM judge to rate the reasoning behind the forecast on five
dimensions: sources, evidence, reasoning synthesis, alignment and uncertainty,
each from 1 to 5. The judge is claude-sonnet-4 at temperature 0
[arxiv:2510.17638 app. C.6] [tab. 3]. The judge rates how a forecast was
reasoned, not whether it turned out right. Its scores agree with 170
human-rated predictions to within a mean absolute difference of 0.37 to 0.44
on a 0-to-4 scale. The main table uses 1 to 5, so the two scales differ
[arxiv:2510.17638 app. C.6.1]. The judge gives GPT-5 (reasoning) 3.88 and
Llama-4-Scout 2.68 on average [tab. 3]. Card: [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](/papers/2510.17638).

### Rule checks and a rubric judge in one weighted score, with a success threshold

KnowU-Bench scores each episode with a rule check and a rubric-conditioned LLM
judge. The paper combines them in a weighted sum with a per-task weight lambda,
set to 1 for fully deterministic tasks, 0 for purely semantic ones, and in
between otherwise [arxiv:2604.08455 §3.4]. An episode counts as a success when
its score is above 0.99 [arxiv:2604.08455 §4.1]. The paper does not name the
judge model. The released code hard-codes its own combination in each task
file, and the judge uses the simulator's model setting (see the card's Paper
vs code section). A judge call that fails scores half credit in the released
code, so API errors can add partial scores (see the card's Pitfalls). A human
check with four raters on 26 trajectories found the hybrid of judge and rules
to have lower error than rules alone; the numbers appear only in a figure
[arxiv:2604.08455 §4.3] [fig. 4]. Card: [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](/papers/2604.08455).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need a per-item verdict and have a gold reference answer | Forced-choice pick against the reference, calibrated on human labels first | It is the only setup with a reference in the corpus [arxiv:2303.12712 §7.2.2]. Agreement with reviewers was 50.8% [arxiv:2303.12712 §7.2.2] |
| You have no reference, and the criterion is a rubric such as fluency or likability | Likert rating with one question per attribute, using a judge that matches the teacher ordering on your data | Only text-davinci-003 and ChatGPT reproduced the teacher ordering; T0 and text-curie-001 did not [arxiv:2305.01937 §3.3] |
| You compare two systems on preference and can afford a validation set | Pairwise preference with a tie option, and check the annotator against humans | GPT-4 agreed with crowd workers on 78% of comparisons [arxiv:2305.11206 §4.1]. The authors call the task subjective [arxiv:2305.11206 §4.1] |
| Human rating is too expensive for the item count | An LLM judge, with a labelled sample to check it | The paper reports US$140 per 200 stories for teachers against under US$5 for the best InstructGPT model. These are 2023 prices and one run [arxiv:2305.01937 §5] |
| The answers differ mainly in length | Check length bias on your data | The GPT-4 card records that the judge tends to reward longer answers [arxiv:2303.12712 §7.2.2] |
| The judge is the model that produced the answers | Use a different judge, or check for self-preference | The GPT-4 judge picked its own answers 87.76% of the time [arxiv:2303.12712 tab. 6]. The paper does not test a fix |
| You need "neither" or "both" options | Add them to the prompt and test them | Humans chose neither 22.75% and both 23.29%, while the judge chose neither 1.23% [arxiv:2303.12712 tab. 6]. Dropping those options from the human labels gives 89.83% agreement [arxiv:2303.12712 tab. 6] |
| Position of the answers may bias the verdict | Judge both orders and report position consistency | In the survey's ten-model table, GPT-4-turbo had the highest position score (80.31%) [arxiv:2411.15594 §4.4]. Swapping order is the check the survey uses [arxiv:2411.15594 §4.4] |
| You want more stable verdicts from one judge | Majority over several rounds, measured on your data | On GPT-3.5-turbo it left alignment unchanged (54.68% against 54.72%) and moved position consistency only slightly [arxiv:2411.15594 §4.4] |
| You need a small open model as judge | Measure it on your data first; Qwen2.5-7B-Instruct was the best open model in the survey's table | Qwen2.5-7B-Instruct scored 56.54% alignment and 63.50% position; Llama3-8B scored 50.72% and 38.85% [arxiv:2411.15594 §4.4]. T0 (13B) and text-curie-001 gave no clear preference in the rating study [arxiv:2305.01937 §3.3] |
| The judge will classify simulated behaviour inside a loop | Use the judge's normalized probability as the paper defines it, and validate it | The paper's threshold is 0.5 on a normalized probability [arxiv:2208.10264 §2] [arxiv:2208.10264 app. F.1]. The released code uses the raw value; see the card [arxiv:2208.10264 §2] |
| You will count categories in a corpus with an LLM | Label a human-coded sample and report agreement before comparing groups | The ICLR category tests are not corrected for multiple comparisons, and the card records no judge agreement [arxiv:2603.18161 tab. 1] |
| Subjective answers have teacher-written criteria | Give the judge the criteria, and check question-level agreement on your data | With criteria, GPT-4-0613's Spearman was 0.854, against 0.820 without them [arxiv:2305.12474 tab. 6]. The criteria are not in the repository, so the numbers cannot be reproduced from it [arxiv:2305.12474 tab. 6] |
| A safety label must be right, and one judge may be unreliable | Vote across several judges, route splits to a person, and report pairwise agreement | Omni-SafetyBench uses five judges and sends 3-2 splits to manual review. Pairwise agreement was 87.5% to 90.4% across the three labels [arxiv:2508.07173 §2.3.1] [tab. 10] |
| The judge's verdict is the training reward | Keep a human-checked sample, and report the judge's error rate with its assumptions | DeepTravel's 89% judge accuracy relies on an assumed 7% human error rate, and the same judge does the evaluation [arxiv:2509.21842 tab. 3] [§5.2.1]. The paper does not test an independent grader |
| There is no programmatic check for a task, and you can score its trajectories with a rubric | Calibrate the rubric on human preference pairs, publish its text, and report the threshold's effect | SIMA 2's reward model scored 0 to 100 with success at 50 or more, calibrated on a small human set. The rubric text and the sensitivity to the threshold are not reported [arxiv:2512.04797 §4.5] |
| Agent runs are judged, and a rule check also exists | Report both signals, and state which disagreement definition you used | In OpenAgentSafety the judge's unsafe rate was above the rule check's for six of seven models, and the disagreement rate ran from 5.69% to 14.95% [arxiv:2507.06134 tab. 3]. The paper calls the unsafe rates conservative lower bounds [§4.3] |
| The judge's model family matches a model under test | Use a judge from another family, or report results split by family | CHI-Bench's judge is Claude Opus 4.7, the same family as several evaluated Claude configurations, and the paper says this may bias results [arxiv:2605.16679 app. F.3] |
| You compare prompts or models by score gaps on open-ended answers | Measure each item under several prompt rewrites and report the average gap | ProSA's values were 0.013 to 0.036 on AlpacaEval and 0.249 to 0.266 on Arena Hard Auto, and the paper says the two benchmarks cannot be compared directly [arxiv:2410.12405 tab. 2] |
| You must label many texts with a codebook and cannot afford annotators for all of them | Zero-shot labels with the human codebook, scored on a sample labelled by trained coders | ChatGPT was about 25 points above MTurk against trained annotators on average, but relevance accuracy ranged from 59% (2023 tweets) to 83% (Congress tweets) [arxiv:2303.15056 §2] [fig. 1] |
| A missed jailbreak is costly, and a false alarm is cheap | Measure both error rates on your labelled set before picking a judge | PAIR's GPT-4-0613 had the highest agreement (88%) and the lowest false-negative rate (7%), with a 16% false-positive rate; Llama Guard had 76%, 47% false negatives and 7% false positives [arxiv:2310.08419 tab. 1] |
| You test whether feedback or answers shift with the user's stated opinion | Pairwise positivity against a baseline reply, after validating the judge on your data | The paper's GPT-4 judge has no reported accuracy, and its grader temperature is not stated for the feedback step [arxiv:2310.13548 app. A.3] |
| Your grader returns free text that you parse for a verdict | Test the parser on sample outputs, and do not rely on a substring test | The notebook scorer counts any output without the word INCORRECT as correct (see the card's Paper vs code section) [arxiv:2310.13548 §3.3] |
| You label answers as supported, partly supported or unsupported against a reference | Start with one holistic rubric that names completeness, and compare it with claim-splitting on a labelled sample from your data | On ASQA the holistic design beat candidate-side splitting for all four judges, at 65 to 81 fewer tokens per call [arxiv:2603.28005 §4.1] [tab. 3]. Reference-side splitting was most accurate on ASQA at 28 to 29% more tokens than the holistic design [arxiv:2603.28005 §4.2]. The author's own labels reversed the ASQA ranking [arxiv:2603.28005 §6.1] |
| You grade agent diagnoses or root causes against a known answer | Write the checklist as yes/no questions, have an expert label a sample, and compare several judges on it before picking one | Claude Sonnet-4.6 reached kappa 0.90 against one expert, GPT-5.4 0.70 and Kimi K2.5 0.76, so the judge choice moves scores. The paper's 7/9 threshold and the code's 0.70 differ [arxiv:2605.07161 §2.5] [tab. 2] |
| An artifact must be judged by its behaviour, and a hidden checklist says what to look for | Replay the behaviour, sample frames, score each requirement, and repeat the judge on fixed evidence before trusting differences | GameCraft-Bench's ten repeated runs had standard deviations of 0.0036 to 0.0050, but the human check was preliminary (22.69 against 26.02) and the paper treats it as calibration only [arxiv:2606.17861 §5.2] [tab. 5]. Sampled frames can miss a flicker shorter than 0.5 s [arxiv:2606.17861 §5.2] |
| You score plans or designs that no program can check, and you can afford several judges | Use judges from different model families, report their agreement, and get expert review on a sample | CellForge's five judges had inter-judge correlations of 0.88 to 0.93, and the paper says experts' scores correspond strongly with them, with no figure given [arxiv:2508.02276 fig. 5] [§ Discussion]. Inter-judge agreement does not show agreement with experts |
| You need a bulk screen of papers and have past human decisions to check it against | A reviewer ensemble with reflection, and a score threshold chosen on your decisions | The calibrated GPT-4o reviewer reached balanced accuracy 0.65, against 0.66 for the NeurIPS human baseline, which used different data [arxiv:2408.06292 tab. 1]. Its threshold moved the zero-shot false-positive rate from 0.11 to 0.38 and the false-negative rate from 0.67 to 0.36 [arxiv:2408.06292 tab. 1] |
| You rank translation systems against human references and do not need segment-level decisions | Zero-shot GPT segment scores with the reference, averaged per system, after checking on your data | System-level pairwise accuracy was 89.8% against 85.0% for MetricX XXL [arxiv:2302.14520 tab. 2]. Segment-level results were weaker than the best metrics [arxiv:2302.14520 tab. 4] [§4.4]. The paper covers three language pairs only [arxiv:2302.14520 Limitations] |
| No reference translation is available | The no-reference variant, checked against your own labels | Without a reference the system-level figure was 87.6%, against 78.8% for COMETKiwi without a reference [arxiv:2302.14520 tab. 2]. Segment-level tau-b fell to 0.31 on en-de [arxiv:2302.14520 tab. 4] |
| The model is small or an older open model | Use a GPT-3.5-class or larger model, and count invalid replies on your data | Babbage and Curie reached 39.1% and 54.4% against 89.8% for GPT-4, and the paper says GPT-2 and Ada gave no usable scores [arxiv:2302.14520 tab. 3] [§4.3]. Its 2023 model ids may no longer behave the same way [arxiv:2302.14520 tab. 1] |
| Several judging prompts are possible for the same rubric | Compare two or three prompt styles on your data, and pick the one you can validate | Stars and no-reference Classes reached 91.2%, above the DA headline of 89.8%, and the paper does not explain its choice of DA [arxiv:2302.14520 tab. 3] [§4] |
| A single judge's verdict is unreliable and you can afford far more tokens | A debate between proposals with a judge, validated on your own benchmark | The full MAC pipeline used 16 to 25 times the tokens of standard baselines, and a single-agent variant was about half the cost and competitive on most datasets [arxiv:2407.15073 §4.4]. The card records no judge-level agreement (not stated) |
| You compare open-ended answers with no gold answer, and you can write criteria for your task | Head-to-head win rates from an LLM judge on those criteria, judged several times per pair, plus a second check | Comprehensiveness win rates were 72% to 83% on podcast answers, but the judge and the claim-based check agreed on only 78% of non-tied comprehensiveness comparisons [arxiv:2404.16130 §5.1] [§5.2] |
| You have several candidate answers per query and can label pairs to train a ranker | A trained pairwise ranker to pick the answer, with an LLM judge used only to score the result | LLM-Blender's fused output reached GPT-Rank 3.01 against 3.90 for the best single model, with the ranker trained on BARTScore labels [arxiv:2306.02561 tab. 2] [§5.1] |
| A generated image, audio or mixed output has criteria that can be checked | Split each check into criteria, ask a vision-language model one question per criterion, and use a program where one exists; validate on your own labelled sample | MMMG reports 94.4% overall human agreement, but the human check covers 41 tasks only, and its judge thresholds are not stated [arxiv:2505.17613 §1] [§3.2] |
| An agent action must be allowed or refused before it runs | Check each parameter with a judge against the agent's stated reason, add a deterministic reference check, and budget for the extra calls | The guarded DREAMS run used about 13 times the input tokens of the unguarded one, and 100% of its steps succeeded against 81% [arxiv:2507.14267 tab. 6]. The guard fails open on infrastructure errors [arxiv:2507.14267 Methods: Safety guard] |
| You want to know whether a model reasons well, not only whether it is right | Rate the reasoning on named dimensions, and compare the judge with human ratings on a sample from your data | LLM-as-a-Prophet's judge differed from 170 human-rated predictions by 0.37 to 0.44 on average, on a 0-to-4 scale [arxiv:2510.17638 app. C.6.1] |
| An episode score mixes rule checks with a judge, and passes only above a high bar | Fix the weights in one place, name the judge model, and report judge errors apart from rule errors | KnowU-Bench counts an episode as a success only above 0.99, and its judge model is not named in the paper [arxiv:2604.08455 §3.4] [§4.1] |

## Implementation sketch

Language-agnostic pseudocode of the forced-choice probe. The output format
follows the paper's prompt [arxiv:2303.12712 §7.2.2]. Lines marked as
suggestions are ours. The Likert variant swaps the pick for a first-number parse
[arxiv:2305.01937 app. D.2.1].

```pseudocode
# Inputs: a gold statement and two candidate answers [arxiv:2303.12712 §7.2.2]
def judge(gold, option_1, option_2):
    # Ask for a similarity and difference note per option before the verdict
    prompt = build_prompt(gold, option_1, option_2)
    reply = llm(prompt)
    # Expect EXPLANATION1, EXPLANATION2, then "BEST: OPTION1" or "OPTION2"
    verdict = parse_best(reply)
    # The probe forces a pick; the paper does not say how invalid replies are
    # handled, so a retry or an explicit failure is needed here (not stated)
    if verdict is None:
        raise JudgeFormatError(reply)
    # Keep the justification with the verdict for audit
    return verdict, reply
```

Suggestion, not from the paper: swap the two options and judge again, then
count a verdict only when both orders agree. The card's example prompt puts the
model's own answer in OPTION1 [fig. 7.5], and the paper does not say whether
the option order was varied. The survey's position-consistency metric is the
same check, with verdicts mapped back to the response they name
[arxiv:2411.15594 §4.4].

## Contradictions

Judge-to-human agreement differs between the papers, and the tasks and metrics
differ too. Both sides are cited; we do not resolve the gap.

- Forced choice on TruthfulQA (GPT-4 against two reviewers): agreement 50.8%.
  The paper calls the subset small and does not check the reviewers' own
  agreement [arxiv:2303.12712 §7.2.2].
- Pairwise preference (GPT-4 against crowd workers and authors): 78% and 79%
  [arxiv:2305.11206 §4.1]. Crowd workers agreed with each other on 82%.
- Pairwise human preference on LLMEval² (survey, ten judges): alignment from
  50.72% to 61.66%, with GPT-4-turbo at 61.54% [arxiv:2411.15594 §4.4]. That
  benchmark and metric differ from the two rows above, so the figures are not
  directly comparable.
- Aggregation: the survey's card summary calls five-round majority voting with
  position swapping among the most useful strategies. Its strategy table shows
  little change in alignment for majority voting on GPT-3.5-turbo (54.68% against
  54.72%) and lower scores than the base judge on two bias columns (nested
  instruction and familiar knowledge) [arxiv:2411.15594 §4.4].
- Question-level correlation (GAOKAO, Spearman 0.825 to 0.864) and
  pass-level agreement (DeepTravel, 82% of checked samples) measure different
  things. They are not comparable with each other or with the rows above
  [arxiv:2305.12474 tab. 4] [arxiv:2509.21842 tab. 3].
- Agent-trajectory judge against humans (OpenAgentSafety): the text reports 94%
  agreement on 100 GPT-4o trajectories with two annotators [arxiv:2507.06134
  §4.3]. Its table 4 lists GPT-4.1 with 39.1% overall disagreement against human
  labels, and the caption does not define the disagreement columns
  [arxiv:2507.06134 tab. 4]. We do not reconcile the two.
- System-level accuracy (GEMBA, 87.6% to 91.2%) scores how well a metric orders
  whole systems against MQM. It is a different measure from the pair-level
  agreement rows above, so the figures are not comparable [arxiv:2302.14520
  §3.2] [arxiv:2302.14520 tab. 2].
- Gilardi's intercoder agreement (91% to 97%) is agreement between two responses
  from the same ChatGPT setting. It is not agreement with human labels, so it is
  not comparable with the agreement figures above [arxiv:2303.15056 §2].
- PAIR's text says Llama Guard has the lowest false-positive rate, but its
  Table 1 lists a BERT classifier at 4%, against Llama Guard at 7% and GPT-4-Turbo
  tied at 7% [arxiv:2310.08419 §3.4] [tab. 1].
- Support-label design ranking (reference-grounded judges): on the constructed
  labels the holistic design beats candidate-side splitting for all three
  primary judges on ASQA. On one author's 60 rows, candidate-side splitting
  beats the holistic design (54 against 52, 54 against 53, 55 against 53
  correct), and reference-side splitting is last. The construction and author
  labels agree on 51 of 60 rows [arxiv:2603.28005 §6.1] [tab. 4]. The paper
  does not say which label standard is right.
- Correctness agreement in the simulated-student study: human against the
  metric is 0.6891, human against the LLM label is 0.7219, and two human
  evaluators agree at 0.7187 [arxiv:2601.04025 tab. 3]. The card does not
  define the "LLM label" row beyond its name, so we do not reconcile the rows.
- Reviewer baseline (AI Scientist): the 0.66 human balanced accuracy comes from
  NeurIPS 2021 data, and the LLM rows use ICLR 2022 papers. The table is not a
  like-for-like comparison [arxiv:2408.06292 tab. 1].
- Harm flag (Omni-SafetyBench): the reported human accuracy of 89.6% on 500
  responses implies about 52 disagreements, while the text reports 39 harmful
  disagreement cases [arxiv:2508.07173 §3.3] [tab. 12].
- Overall agreement (MMMG): the abstract gives 94.4% human agreement. The
  per-modality figures in the introduction average to about 93.4% by the
  card's arithmetic [arxiv:2505.17613 §1]. We do not reconcile the two.

On the story rating task, the absolute scores also depend on the judge.
text-davinci-003 scored higher than the teachers, and ChatGPT scored lower, so
systems are only comparable when rated by the same judge [arxiv:2305.01937 §3.3]
[arxiv:2305.01937 tab. 1].

## Papers

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](/papers/2303.12712): a forced-choice GPT-4 judge against a gold statement on TruthfulQA, with agreement against two human reviewers.
- [Can Large Language Models Be an Alternative to Human Evaluations?](/papers/2305.01937): reference-free Likert ratings by several LLMs, compared with three certified teachers on story generation.
- [LIMA: Less Is More for Alignment](/papers/2305.11206): GPT-4 as a pairwise preference annotator, with agreement against crowd workers, plus a graded ChatGPT judge in the ablations.
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](/papers/2305.12474): GPT-4-turbo grading subjective exam answers against teacher marking criteria, with question-level correlation to human scores.
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](/papers/2410.12405): judge-scored open-ended answers used to measure how much a model's scores change across rewritten prompts.
- [A Survey on LLM-as-a-Judge](/papers/2411.15594): a taxonomy of judge design, reliability strategies, and a meta-evaluation of ten judge models on human alignment and bias.
- [Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models](/papers/2508.07173): five judge LLMs voting on comprehension, harm and refusal flags, with manual review of 3-2 splits.
- [OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety](/papers/2507.06134): a GPT-4.1 judge that reads agent trajectories, run alongside a rule-based check of the final state.
- [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](/papers/2509.21842): LLM verifiers as the binary reward for training a travel-planning agent, and as the grader of its evaluation.
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](/papers/2208.10264): LM judges classify simulated subjects' actions inside a Milgram-style loop.
- [How LLMs Distort Our Written Language](/papers/2603.18161): gpt-4o as a coder of stance, argument types and review categories in a user study and an ICLR 2026 review corpus.
- [CHI-Bench](/papers/2605.16679): a three-vote rubric judge with a strict majority, combined with a deterministic check to give a binary trial reward.
- [Evaluation and Benchmarking of LLM Agents: A Survey](/papers/2507.21504): a survey of agent evaluation that lists code-based checks, LLM-as-a-judge, agent-as-a-judge and human review as metric options, with no experiments of its own.
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](/papers/2303.15056): zero-shot ChatGPT labels with the human codebook, compared with crowd-workers and trained annotators.
- [Towards Understanding Sycophancy in Language Models](/papers/2310.13548): GPT-4 as a pairwise judge of feedback positivity, and a grader and a mistake check in sycophancy tests.
- [Jailbreaking Black Box Large Language Models in Twenty Queries](/papers/2310.08419): a judge that scores each attack attempt in an attacker-target loop, with a comparison of six judges on agreement and error rates.
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](/papers/2406.12708): GPT-4 codes the reasons for acceptance and rejection in a simulated peer review, alongside the simulation's reviewer agents.
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](/papers/2302.14520): zero-shot GPT scores for each translated segment, with or without a reference, averaged per system on WMT22 MQM.
- [Multi-Agent Causal Discovery Using Large Language Models](/papers/2407.15073): a judge that picks the winner between debate proposals in a causal-discovery pipeline.
- [A Survey on Evaluation of Large Language Models](/papers/2307.03109): a survey of evaluation tasks, benchmarks and methods, with no new experiments; the card's body describes no model-based judge.
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](/papers/2306.02561): a trained pairwise ranker over candidate answers, with ChatGPT pairwise judgments as the evaluation metric.
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](/papers/2404.16130): LLM judge win rates on author-written criteria for query-focused answers, with a claim-based check.
- [Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison](/papers/2603.28005): four single-call judge designs for three-way support labels against a reference, with a holistic rubric compared with claim-splitting on ASQA, QAMPARI and TruthfulQA.
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](/papers/2408.06292): a GPT-4o reviewer with ensembling, reflection and a score threshold, checked against ICLR 2022 decisions as one stage of an automated research pipeline.
- [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](/papers/2512.04797): a Gemini reward model that rates agent game-play videos from 0 to 100 with a success threshold, used in a self-improvement loop.
- [Simulated Students in Tutoring Dialogues: Substance or Illusion?](/papers/2601.04025): a GPT-5 mini correctness and error judge for simulated tutoring students, with human agreement on a subset and the judge also feeding a DPO reward.
- [SREGym](/papers/2605.07161): a yes/no checklist judge that grades agent diagnoses in a live Kubernetes benchmark, validated against one expert.
- [AI Scientists for Building Virtual-Cell Models](/papers/2508.02276): a panel of five judges from separate model families that scores generated plans, with expert review of a subset, in a multi-agent pipeline.
- [GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?](/papers/2606.17861): a multimodal GPT-5.5 judge that scores replayed game videos against a hidden per-requirement rubric, with repeated-run stability and a small human check.
- [MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613): per-criterion questions to vision-language and audio judges, with program checks, validated against 41 tasks of human labels.
- [DREAMS: Density Functional Theory Based Research Engine for Agentic Materials Simulation](/papers/2507.14267): a per-parameter judge guard before each scientific tool call, a report-time judge over provenance, and a token-cost comparison.
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](/papers/2510.17638): a 1-to-5 reasoning-quality judge for forecasts, checked on 170 human-rated predictions, alongside proper scores and market return.
- [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](/papers/2604.08455): a rubric judge mixed with rule checks in a weighted episode score, with a 0.99 success threshold.