Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison

In plain words

Suppose an AI judge has to read an answer next to a reference text and say whether the answer is fully supported, partly supported, or unsupported. One way to ask is to split the answer into small claims and check each claim. Another is to give the judge a checklist and ask for one overall rating. This paper compares these approaches on three question-answering datasets, with several AI models acting as judges.

The paper also tries splitting the reference text into claims instead, so the judge checks whether the answer covers each reference point. This matters when the label is about completeness: checking only the answer's own claims can miss what the answer leaves out.

The main lesson is that the right design depends on what the label is supposed to measure. On a completeness label, splitting the reference worked best, but it used more tokens per call. On a label about contradicting wrong statements, splitting the answer did as well as or better than a single rating. A small check with one author's own labels ranked the designs differently, which shows why you should decide first what the label means.

TL;DR

Compares four single-call judge designs on three-way support labels (supported, partially supported, unsupported) built from TruthfulQA, ASQA and QAMPARI, using four LLM judges. Splitting the candidate answer into claims is the weakest design where labels reward completeness, and the holistic rubric is cheaper and more accurate on ASQA and QAMPARI §4.1. Splitting the reference is the most accurate on ASQA, at about 28-29% more tokens than the holistic rubric §4.2. All results are author-reported and measured against constructed labels, and a single-author check reverses the ranking §6.1.

Problem

Splitting a generated answer into atomic facts and checking each one is a known factuality method, used in long-form pipelines such as FActScore §1 §2. In a single judge call, the usual version decomposes the candidate. A support label, however, mixes factual precision (is each claim backed by the reference?), completeness (does the answer cover the reference?), and audit artifacts such as claim lists. Candidate-side decomposition targets precision and produces artifacts, but its structure does not list what the candidate leaves out §1.

The paper studies this in a setting where the reference is supplied in the input, and it treats the judging instruction and its token cost as experimental variables §2 §3.3.

Method

Four single-call designs share the judge model, the inputs and the layout of the user message. Each call returns one JSON object whose first key is verdict §3.2 fig. 1 app. G.

1. Labels. Each source question yields two rows. The supported row is the reference itself, copied verbatim. On TruthfulQA the other row is an incorrect benchmark answer, usually a common misconception, labelled unsupported. On ASQA the partial row renders the first half of the disambiguated qa_pairs as "sub-question + first short answer". On QAMPARI, the partial row keeps the earlier half of the gold answers, joined by commas §3.1 tab. 1 app. H. Partial labels encode coverage of the reference, not factual error §3.1.
2. Design C (candidate-side atomic). The judge splits the answer into claims and checks each one against the reference. Its v1 prompt does not mention completeness §3.2.
3. Design H (matched holistic). The judge works through a four-point checklist (correctness, completeness, unsupported detail, style robustness) and is told to skip claim-by-claim splitting §3.2.
4. Design R (reference-side atomic). The judge splits the reference into claims and checks whether the candidate covers each one. It is not asked about candidate content outside the reference §3.2.
5. Design B (bidirectional). The judge does both, returning six claim lists §3.2 app. G.
6. Scoring. Only the parsed verdict is scored. Accuracy is computed over 400 rows per dataset §3.1 §3.4.

The outputs in the primary runs also carry score (integer 1-5), confidence and rationale. These do not enter accuracy app. G.

# One judge call per row; one design per call  [§3.2] [fig. 1]
# question, reference, candidate: strings  [§3.1]
def judge_row(design, question, reference, candidate):
    # Shared user message: question, reference, candidate, then the JSON keys  [app. G]
    # Design C: split candidate into claims; ground each in the reference  [§3.2]
    # Design H: holistic rubric incl. completeness; no explicit decomposition  [§3.2]
    # Design R: split reference into claims; check each is covered by candidate  [§3.2]
    # Design B: both directions; six claim lists  [§3.2]
    prompt = build_prompt(design, question, reference, candidate)
    # Temperature and output cap were not set in the primary runs  [app. I]
    output = judge_model(prompt)
    # Parse whole JSON, then embedded JSON, then regex fallback  [app. H]
    return parse_verdict(output)

# Exact sign test at the source-question level  [§3.4]
# rows_a, rows_b: correct rows per question (0-2) for two designs  [§3.4]
def sign_test(rows_a, rows_b):
    # Count questions where one design gets more rows right  [§3.4]
    wins_a = count(a > b for a, b in zip(rows_a, rows_b))
    wins_b = count(b > a for a, b in zip(rows_a, rows_b))
    # Ties are excluded; two-sided binomial test; no multiplicity correction  [§3.4]
    return exact_binomial_two_sided(wins_a, wins_a + wins_b)

Prompt-controlled comparison. Designs are compared under matched inputs and comparable instruction detail, not matched compute. Output length differs by design, and realized tokens are reported as an outcome §3.3. The holistic rubric names completeness explicitly, while C v1 does not. C v2 asks for "factual accuracy and completeness" to test this asymmetry §3.3 §4.4.

Diagnostics. Schema-matched control, output caps (128, 256 and 512 tokens), reference truncation and removal, perturbations, and prompt variants are run on 40-50 row slices, mostly with Opus and GPT-4.1 §5 app. A app. B. The paper states these measure sensitivities, not mechanisms §5.

Hyperparameters

Values not stated in the paper are written "not stated". Judge settings come from the paper's appendix, not from code.

Results

Author-reported. Setup: three QA datasets with constructed labels, 200 questions (400 rows) per dataset, judges as listed §4 tab. 2. Accuracy is row-level agreement with the construction labels. Tokens are mean total (input plus output) tokens per call.

Source: tab. 2. The most accurate design per dataset and judge is in bold in the paper.

Candidate-side versus holistic (ASQA, QAMPARI, TruthfulQA). On ASQA, H is 7.25 to 15.5 points more accurate than C for all four judges and uses 65 to 81 fewer tokens per call. The paired sign tests are significant at p ≤ 3×10⁻⁸ at question level §4.1 tab. 3. Partial-row accuracy gives the gap: C recognises 55.5%, 16%, 12.5% and 43% of partial rows for Opus, GPT-4.1, Gemini and Sonnet, against 70%, 46.5%, 45.5% and 62% for H tab. 15. On QAMPARI H is 1.75 to 7.0 points ahead of C (p ≤ 0.016) and cheaper, because C's outputs list long answer sets §4.1 tab. 16. On TruthfulQA, C is at least as accurate as H for every judge. The difference is significant for Gemini and Sonnet (12:0 questions in C's favour, p = 4.9×10⁻⁴), and is not significant for Opus and GPT-4.1 §4.1 tab. 3.

Reference-side and bidirectional (ASQA). R is the most accurate ASQA design for each judge it was run with: 0.975, 0.9325 and 0.8675 for Opus, GPT-4.1 and Gemini. That is 12.5, 21.25 and 17.0 points above H, and R wins against H on 50:0, 84:1 and 69:2 questions (p ≤ 1.8×10⁻¹⁵). The token cost is 28-29% more than H §4.2 tab. 17. On QAMPARI R stays close to H (0.995-0.9975 vs 0.9925-0.9975) at 63-69% more tokens. On TruthfulQA R stays within 2.5 points of the other designs §4.2. B improves on H for ASQA but loses to R on 30-54 questions and costs 71-74% more tokens than H §4.2.

Judge pattern. Across the three primary judges, H minus C accuracy is positive in 21 of 24 complexity-bin cells and zero in the other three, all on QAMPARI. The ASQA gap shows no monotone trend across complexity bins §4.3 tab. 19.

Fourth judge and prompt variants. Sonnet-4.6 gives the same pattern: H beats C on ASQA (0.810 vs 0.715) and QAMPARI (0.995 vs 0.9625), and C beats H on TruthfulQA (0.975 vs 0.945) §4.3. On ASQA slices, every holistic prompt variant beats every candidate-side variant for Opus (0.88-0.94 vs 0.78) and Sonnet (0.88 vs 0.72-0.76), including C v2 which asks for completeness explicitly §4.4 tab. 7.

Diagnostics (slices, not primary results). A schema-matched judge that returns C's keys without the decomposition instruction scores 0.800 (Opus) and 0.620 (GPT-4.1) on a 50-row ASQA slice, below H on the same rows §5 tab. 8. With output caps of 128, 256 and 512 tokens, the ASQA ordering stays the same (C about 0.76 vs H 0.86-0.88 for Opus) §5 tab. 9. Removing the reference lowers TruthfulQA accuracy to 0.55-0.75 but leaves it unchanged under a truncated reference. ASQA drops 20-24 points under truncation §5 tab. 10. Swapping in another example's reference flips 34-50% of ASQA verdicts; TruthfulQA flips stay at or below 5% §5 app. A tab. 5.

Author labels (60 rows, one author). Against the construction labels, H beats C for all three judges. Against the author's labels, C beats H for all three (54 vs 52, 54 vs 53, 55 vs 53 correct), and R is last. The construction and author labels agree on 51 of 60 rows, including all 40 TruthfulQA and QAMPARI rows and 11 of 20 ASQA rows. The nine disagreements are ASQA rows the construction labels call partially supported and the author calls supported §6.1 tab. 4 tab. 13 tab. 14.

Generated ASQA answers. On 200 rows from 100 answers by two generators, pooled, the holistic design gets more rows right than candidate-side for each judge: 95 against 88 (Opus), 113 against 76 (GPT-4.1), and 123 against 103 (Gemini). R labels 16-38% of full answers correctly but 74-78% of nominal partials, which the paper reads as R flagging most generated answers as incomplete. The paper treats this set as a noisy stress test, not a factuality benchmark §6.2 tab. 11 tab. 12.

Paper vs code

No official code.

Search: the arXiv abstract page has no code link and no code availability statement (its comment reads "Accepted to AACL-IJCNLP 2026"). A GitHub repository search for llm_as_judge_lab, the prompt module path named in the appendix app. G, returned only unrelated repositories. Searches for llm judge atomic holistic reference support and for the author name with "Berkeley" in GitHub users returned no matches. Prompts and settings on this card therefore come from the paper alone.

Pitfalls

• Constructed labels measure coverage, not truth. Partial labels are deterministic and lexical, and an ASQA partial row may contain only accurate claims and still carry the partial label §3.1. The paper calls them construction labels and says the scores measure agreement with a strict reference-completeness standard §3.1 §8. Do not read the ASQA and QAMPARI accuracies as factuality accuracy (Limitations).
• The ranking flips with the label standard. The author's labels reverse the ASQA ranking of C and H, and R is last under them §6.1. Decide what the label should mean before choosing a design §7.
• Supported rows are easy. Every design scores 0.94-1.00 on supported rows, so the discriminating class is the partial or unsupported one §3.1 tab. 15.
• Parsing is not strict JSON. Of 7,200 primary C and H outputs, 5,686 are whole JSON objects, 1,512 need extraction from embedded text and 2 need regex recovery. All verdicts are recoverable, but this does not show format compliance app. H tab. 20.
• Decoding is not fixed. Temperature and output caps were left at provider defaults in primary runs. The paper could not resolve the model snapshots behind its aliases, so reruns may not match app. I (Limitations).
• Output caps differ across runs. Opus C and H in the generated-answer set carry a 256-token cap, while other judging calls there have no cap. Generation settings were not logged per call app. C.
• The holistic prompt is not symmetric with C v1. H names completeness and C v1 does not, so part of the gap may come from prompt wording. C v2 adds completeness and still trails H on ASQA, but the paper calls the slices small and reads them as an ordering, not an effect size §3.3 §4.4.
• Reference quality matters a lot on ASQA. Changing the reference changes ASQA verdicts far more than TruthfulQA ones §5 app. A. Check the reference first.
• Generated-answer labels are noisy. 25 of 100 nominal partials equal the full answer plus a period, and Opus judges its own 50 generations §6.2 app. C. The paper states it cannot rule out self-preference.
• Significance is unadjusted. Sign-test p-values have no multiple-comparison correction. The bootstrap and the sign test can disagree: for TruthfulQA GPT-4.1 the 95% interval excludes zero but p = .125 §3.4 app. E.
• Token counts are not cost. Totals are tokens per call, not dollars or latency, and Gemini has no output-token breakdown app. E app. I.
• Sonnet-4.6 is partial. It was run only with C and H, and budget and grounding diagnostics mainly use Opus and GPT-4.1 (Limitations) app. B.
• Diagnostics are small. Most diagnostic slices have 40-50 rows from 20-25 questions §5 app. B. Treat differences of a few points as noise.
• Claim lists are not scored. Accuracy uses only verdict. A judge's claim inventory is an artifact, not evidence of better labels, and the paper says it should not be assumed to raise accuracy §7.

Lineage

• Builds on: atomic-fact verification of long-form generations, FActScore §1 §2. Also related: SAFE (multi-step decomposition with search), VeriFact (reference fact sets for recall), FaStFACT and UFO §2.
• Compared against: the matched holistic rubric (H), with the bidirectional design (B) as an added variant §3.2.
• Relates to: reference-guided rubric judges such as F²Bench §2.
• Superseded by: no card in this wiki.