wikillm

Paper · 2026-11-12 · cs.AI, cs.LG

SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0

SAGE-QA adds two LLM-judge checks to SQuAD 2.0 exact match, for answers that fail it: reference equivalence (SES) and passage support (RVS). Zero-shot Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM; LoRA/QLoRA fine-tuning adds 21 to 34 EM points to open models. Judge scores, author-reported: SES 92.81 and 91.67, RVS 97.96 and 98.02.

LLM-as-a-judgeLLM evaluationExtractive question answeringLow-rank adaptation (LoRA and variants)question-answeringevaluationllm-as-a-judgesquad-2.0fine-tuning

arXivPDFMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A reading-comprehension benchmark gives a model a short passage and a question. The model must answer from the passage alone, or say that the passage does not contain the answer. The usual score checks whether the answer matches a reference string after the text is tidied. A correct answer worded differently from the reference therefore scores zero, and modern chat models often word their answers differently. Their score then falls for wording, not for mistakes.

The paper keeps the string score and adds two checks. A second language model acts as a judge, and it runs only on answers that fail the string check. The first check asks whether the answer means the same as a reference answer. The second asks whether the passage supports the answer and whether the answer actually addresses the question, without showing the judge any reference. The judge's grades are combined with full credit for the exact matches, which gives scores that show where the string rule and the judge disagree.

The authors also test open-weight models before and after light fine-tuning on SQuAD training data. The fine-tuned models score much higher on the string measure. The main lesson is that one number hides the difference between a paraphrase and a wrong answer, so the judge results should be read next to it.

TL;DR

SAGE-QA is an evaluation framework for SQuAD 2.0 that keeps exact match (EM) and a count-based F1, then sends each failed answer to one external judge model. The judge scores reference equivalence (SES) and passage support (RVS) §III.C §III.D. On a fixed 10,000-item validation subset, zero-shot Claude Opus 4.6 and Gemini 3.1 Pro reach 85.21 and 86.66 EM, and the judge gives dataset-wide scores of 92.81 and 91.67 (SES) and 97.96 and 98.02 (RVS) tab. V. These judge scores are diagnostics, not correctness rates. Use this card if you need a graded, reference-aware or passage-aware evaluator for closed-context QA, and read the pitfalls before trusting any number.

Problem

Exact match penalises every difference from the annotator's span or wording, so a valid paraphrase or a different passage-supported span scores zero §I §II.A. The paper's example is a correct answer that is marked wrong only because the references include extra leading or trailing words §II.A.

EM also cannot tell a paraphrase from a substantive error §I. SQuAD 2.0 adds unanswerable questions, so a system must also abstain. An answer given where the benchmark expects an abstention counts as a false positive, but EM alone does not show whether that answer is supported by the passage §I §III.A.

Assumptions: the reference answers are treated as ground truth §I; the judge must be a model that generates none of the evaluated answers §I §IV.C; the SES and RVS results are judge-based diagnostics and not measured correctness or hallucination rates §VI.B.

Method

The framework has three stages: a lexical stage that assigns outcomes, two judge diagnostics on routed pools, and aggregation §III. For each item i, the inputs are the passage c_i, the question q_i, the reference set G_i and the answerability label a_i in {0, 1} §III.A.

Step 1: abstention flag. Set z_i = 1 when the raw answer is null, blank, an empty list or the literal []. Otherwise set z_i = 0. The interpreted prediction ŷ_i is blank for abstentions and holds the extracted answer for everything else. The flag is read before any normalisation and is never recomputed from normalised text §III.A.

Step 2: exact match. Normalise prediction and references for the EM comparison only: lowercase, remove articles and punctuation, collapse whitespace §III.A. Each item gets e_i = 1 if the normalised prediction equals a normalised reference, and EM is 100 times the mean of e_i eq. 1. For an unanswerable item the normalised reference set is the single empty string §III.A.

Step 3: outcome partition. Each item gets one outcome: TP (answerable, not abstained, exact match), TN (unanswerable, abstained), FP (unanswerable, answered) or FN (answerable, and either abstained or answered without an exact match). FN splits into FN_z (false abstentions) and FN_a (answered non-matches) §III.A §III.B eq. 2.

Step 4: counts and metrics. On the 10,000-item split, TP + FN = 4,990 and TN + FP = 5,010 eq. 3. EM equals 100 (TP + TN) / N eq. 4. Precision, recall and decision-matrix F1 follow from the counts eq. 5 eq. 6. The false-positive rate is 100 FP / (FP + TN) and the false-negative rate is 100 FN / (TP + FN) eq. 7.

Step 5: SES pool and score. The SES pool is C_S = FN_a eq. 8. The judge sees only the reference alternatives and the answer eq. 9 §III.C. Each reference is scored separately and the maximum is kept. The five bands are: 1.00 for mutual equivalence with no material omission or addition; 0.95 to 0.99 for strong equivalence with only minor, non-essential differences; 0.80 to 0.94 for partial equivalence with a meaningful omission or addition; 0.20 to 0.79 for weak agreement with major divergence; 0.00 for unrelated or contradictory meaning §III.C. Items outside the pool get 1.0 for TP and TN and 0.0 for FP and FN_z alg. 1.

Pseudocode
# Graded reference-meaning score for one item  [alg. 1] [§III.C] [eq. 9]
def ses_score(outcome, abstained, prediction, references):
    if outcome in ("TP", "TN"):
        return 1.0   # automatic full credit, no judge call
    if outcome == "FP" or abstained:
        return 0.0   # FP and false abstentions get no credit
    # One judge call per reference; keep the best  [§III.C]
    scores = [judge_ses(ref, prediction) for ref in references]
    return max(scores)   # must lie in [0, 1]; resubmit otherwise  [alg. 1]

Step 6: RVS pool and score. The RVS pool is C_R = FP ∪ FN_a eq. 12. The judge receives only the passage, the question and the answer eq. 13. It first decides from the passage alone whether the question is answerable and what evidence the passage gives. It then scores the answer for groundedness (every claim stated or clearly implied by the passage) and completeness (the answer fully addresses the question). It uses the same five bands as SES §III.D. The binary acceptance r^τ is 1 when the score is at least τ = 0.95 §III.D. The judge receives no reference answers and no answerability label §III.A.

Pseudocode
# Passage-support score for one item; no references are passed  [alg. 2]
# [§III.D] [eq. 13]
def rvs_score(outcome, abstained, passage, question, prediction):
    if outcome in ("TP", "TN"):
        return 1.0   # automatic full credit, no judge call
    if abstained:
        return 0.0   # false abstentions sit outside the pool
    return judge_rvs(passage, question, prediction)   # in [0, 1]

Step 7: aggregation. Each diagnostic reports a case mean over its pool eq. 10 eq. 14 and a dataset-wide score that credits TP and TN in full eq. 11 eq. 15. The dataset-wide score equals EM plus the pool share times the case mean. A binary RVS aggregate counts TP and TN as credited, plus the pool items with r ≥ 0.95, over all N items §III.D.

Pseudocode
# Dataset-wide score: exact matches get full credit  [§III.C] [eq. 11]
def dataset_wide_score(num_tp, num_tn, pool_scores, n_items):
    total = num_tp + num_tn + sum(pool_scores)
    return 100 * total / n_items   # 0-100 score, not a share of answers
Pseudocode
# Decision-matrix F1 from integer outcome counts  [§III.B] [eq. 6]
def decision_f1(tp, fp, fn):
    return 100 * 2 * tp / (2 * tp + fp + fn)

The same external judge scores both diagnostics, but in two separate requests with different inputs §III.A §IV.C. Neither request includes the benchmark answerability flag §III.A.

\mathrm{SES}_{\mathrm{all}} = \mathrm{EM} + \frac{|\mathcal{C}_S|}{N}\,\mathrm{SES}_{\mathrm{cases}} \qquad [\mathrm{eq.}\ 11]

Hyperparameters

Values are from tab. III and §IV.A–§IV.C. "Not stated" means the paper does not give the value.

NameValueWhereNotes
LoRA rank, Llama 3 8B128tab. IIIα = 32; LoRA via Unsloth §IV.C
LoRA rank, gpt-oss-20b128tab. IIIα = 32; 4-bit quantised base with LoRA §IV.C
LoRA rank, Llama 3.3 70B32tab. IIIα = 64; QLoRA with 4-bit NF4, BF16 compute §IV.C
learning rate5e-5 (8B), 3e-5 (gpt-oss), 1e-5 (70B)tab. IIIschedule stated for 70B only §IV.C
batch size32 (8B), 64 (gpt-oss), 60 (70B)tab. III70B value is effective, via gradient accumulation
max sequence length512 (8B), 512 (gpt-oss), 1,024 (70B)tab. IIItruncation rule not stated
epochs1§IV.Call three open-weight models
optimiser8-bit AdamW§IV.Cbetas, eps not stated
training lossassistant-response cross-entropy§IV.Ctraining prompt template not stated
schedule, warmup, weight decaycosine; 0.03; 0.01§IV.Cstated in the 70B sentence only
LoRA dropout, target modulesnot stated§IV.C
training data87,000 items from SQuAD 2.0 training split§IV.A"balanced" criterion not stated
evaluation data10,000 SQuAD 2.0 validation items (4,990 answerable, 5,010 unanswerable)§IV.Aselection method and seed not stated
generation decodingnot stated (temperature, output limit, seed)§IV.Btwo-field JSON output via structured-output schema
prompt for zero-shot runsGEPA-compiled prompt; text not reproduced§IV.Bpilot item count not stated
judge modelgpt-6-luna, medium reasoning effort§IV.Cbatch API; snapshot and sampling not stated
RVS threshold τ0.95§III.D tab. V
judge bands1.00; 0.95–0.99; 0.80–0.94; 0.20–0.79; 0.00§III.C §III.Dmapping of values in the gaps not stated

Pilot prompt selection §IV.B: among three human-written prompts, one OpenAI-optimised prompt and prompts compiled with DSPy MIPROv2 and GEPA, the GEPA prompt scored highest, at 69.60 EM and 80.70 token-F1 on the pilot set. The pilot set's size and source are not stated.

Results

Author-reported. Setup: 10,000 SQuAD 2.0 validation items §IV.A; zero-shot Claude Opus 4.6 and Gemini 3.1 Pro against untuned and fine-tuned Llama 3 8B, Llama 3.3 70B and gpt-oss-20b §IV.A. The untuned open-weight models are the matched baseline for fine-tuning §V.A. Each configuration was run once; no variance is reported.

Outcome counts, EM and decision-matrix F1 tab. IV:

ConfigurationTPFN (total)FN_z / FN_aFPTNEMF1
Claude Opus 4.6, zero-shot3,9371,053165 / 8884264,58485.2184.19
Gemini 3.1 Pro, zero-shot4,220770171 / 5995644,44686.6686.35
Llama 3 8B, untuned3,2361,754107 / 1,6472,5582,45256.8860.01
Llama 3.3 70B, untuned3,5761,414120 / 1,2941,4473,56371.3971.43
gpt-oss-20b, untuned3,6101,380112 / 1,2681,9923,01866.2868.16
Llama 3 8B, fine-tuned4,681309not split5974,41390.9491.18
Llama 3.3 70B, fine-tuned4,684306not split2704,74094.2494.21
gpt-oss-20b, fine-tuned4,039951not split3024,70887.4786.57

Main findings, all §V.A unless noted:

  • Gemini's EM lead of 1.45 points over Claude comes from 283 more correct answers on answerable items, offset by 138 more answers on unanswerable items fig. 2. False abstentions are similar for both §V.A.
  • Untuned open-weight models answer unanswerable items far more often: false-positive rates are 28.88% to 51.06%, against 8.50% and 11.26% for the frontier models tab. IV.
  • Parameter-efficient fine-tuning raises EM by 21.19 to 34.06 points, and lowers both FP and FN counts fig. 4. The fine-tuned 70B abstains on 327 more unanswerable items than the fine-tuned 8B, and this accounts for most of its EM advantage §V.A.

Judge diagnostics for the zero-shot frontier models, τ = 0.95 tab. V:

Metric (0–100)Claude Opus 4.6Gemini 3.1 Pro
Baseline EM85.2186.66
SES pool size, C_S = FN_a888599
SES mean on pool85.5683.68
SES dataset-wide92.8191.67
RVS pool size, C_R = FP ∪ FN_a1,3141,163
RVS mean on pool97.0197.66
RVS dataset-wide (graded)97.9698.02
Share of pool with r ≥ 0.9593.68%92.86%
RVS dataset-wide (binary)97.5297.46
  • On SES, the ranking reverses the EM ranking: Claude scores 92.81 and Gemini 91.67 §V.B.
  • On RVS, the graded gap is 0.06 points, and the binary ordering flips §V.B. The paper says the proximity does not show equal correctness §V.B.
  • The judge's RVS score is at least 0.95 for 385 of Claude's 426 false positives and 517 of Gemini's 564, which the paper summarises as over 90% tab. VI §V.B.

Score bands per pool tab. VI:

Pool (n)1.000.95–0.990.80–0.940.20–0.79<0.20
SES, Claude (888)962194935030
RVS on FP, Claude (426)3731218320
RVS on FN_a, Claude (888)825213066
SES, Gemini (599)871602755225
RVS on FP, Gemini (564)4962131412
RVS on FN_a, Gemini (599)546173051

Paired decisions on the answered non-matches at τ = 0.95 tab. VII:

DecisionClaude Opus 4.6Gemini 3.1 Pro
accepted by both SES and RVS310242
accepted by SES only55
accepted by RVS only536321
accepted by neither3731

SES accepts 35.5% of Claude's and 41.2% of Gemini's answered non-matches, and RVS accepts 95.3% and 94.0% of them §V.B. The judge made 3,964 requests in total, equal to the two pool sizes summed over both models, and every response was a valid score in [0, 1] §IV.C.

Paper vs code

No official code.

Searches: the paper and its abstract give no code link § abstract. GitHub repository search for "sage-qa" returned three repositories with this name (owners lamm-mit, jloe2911 and MkYCHsu), none linked to the authors, and they were not used. A search for "SAGE-QA closed-context" returned nothing, and a search for the author surname "Sekharan" found no repository tied to the authors. User searches for the other two author names found no account tied to the paper.

Pitfalls

  • EM formula and count formula disagree in one case. Eq. 1 counts a non-abstaining answer as exact when it normalises to the empty string, even on an unanswerable item. Eq. 4 counts only TP and TN, and such an item is an FP. The reported EM values match the count form. The paper does not say whether any such item occurred §III.A eq. 1 eq. 4. Implement the count form and log these cases.
  • Abstention is read from the raw response. Null, blank, [] and empty-list answers count as abstentions. An answer that only normalises to empty is not an abstention and is an FP §III.A.
  • Judge request count does not match the text. The text and Algorithm 1 call the judge once per reference and keep the maximum §III.C alg. 1. The reported total of 3,964 requests equals |C_S| + |C_R| exactly §IV.C, which implies one request per eligible item. The paper does not say how references were sent.
  • Judge bands leave gaps. Values such as 0.945, 0.795 or 0.195 fall in no band. The paper does not say how continuous judge output is mapped to the bands. The τ = 0.95 cut sits at the boundary between the 0.94 and 0.95 bands, so acceptance depends on how that edge is handled §III.C §III.D.
  • Judge prompts are not published. Only the rubric and the input fields are described. The RVS two-step instruction (answerability first, then groundedness and completeness) is described in prose only §III.D. Expect different results with different prompts.
  • Generation and judge settings are missing. Temperature, output limit, seed, retry limit and system prompt are not stated for any model. The judge's snapshot and sampling settings are not stated either §IV.B §IV.C.
  • Pools and test set are unexplained. The method for picking the 10,000 validation items is not stated, nor is the balancing rule for the 87,000 training items §IV.A.
  • RVS scores are not correctness. RVS accepts over 90% of the benchmark's false positives, which the paper reads as disagreement with the answerability labels, not as verified correctness §V.B §VI.B. Judge scores also credit exact matches automatically, so compare score points rather than percentages of correct answers §III.C.
  • Introduction and method differ on RVS scope. The introduction says RVS runs on all EM failures. Section III and Algorithm 2 exclude false abstentions, which get 0 with no judge call §I §III.D alg. 2.
  • Single judge, no human check. The paper uses one judge model and no human adjudication or judge ensemble, and it names this as future work §VI.B §VII.
  • Leaderboard comparison is not like for like. Table I values come from the official hidden test set. All numbers in this paper are on 10,000 validation items tab. I §IV.A. The introduction gives a "90.58%–90.94%" prior ceiling, which appears nowhere else in the paper §I. Fine-tuned Llama 3 8B scores 90.94, which equals the leaderboard's 90.939 at printed precision. Only the fine-tuned 70B (94.24) clearly exceeds it tab. I tab. IV.
  • F1 here is not SQuAD F1. Table IV reports decision-matrix F1 eq. 6, not the token-overlap F1 used on the leaderboard §II.A §III.B.
  • Fine-tuning details are partial. The cosine schedule, warmup ratio 0.03 and weight decay 0.01 appear only in the 70B sentence §IV.C. Not stated for the 8B and gpt-oss models: schedule, warmup, weight decay, LoRA dropout, target modules, micro-batch size and training prompt template tab. III.
  • "Rank-stabilized LoRA" is cited to QLoRA. The paper cites the QLoRA reference for rank-stabilized LoRA, and the scaling rule is not given §IV.C [qlora].
  • Fine-tuning gains are in-distribution. Training and evaluation both use SQuAD 2.0 data, and the paper tests only on SQuAD 2.0 validation. Transfer to other datasets is untested §IV.A §VI.B.
  • Judge-band means are not published. Table VI gives counts per band, not band means, so the SES case means in Table V cannot be recomputed exactly from the paper tab. V tab. VI.
  • Single run, no variance. Differences such as the 0.06-point RVS gap are reported without any estimate of run-to-run noise §V.B.

Lineage

  • Builds on: SQuAD 2.0 (Rajpurkar et al., ACL 2018; DOI 10.18653/v1/P18-2124) for the data §II.A. Normalisation follows the Hugging Face SQuAD v2 metric §III.A. Fine-tuning uses LoRA (arXiv:2106.09685) and QLoRA (arXiv:2305.14314) §IV.C. The zero-shot prompt comes from DSPy MIPROv2 and GEPA prompt compilation §IV.B.
  • Judge methods named as related work: G-Eval (arXiv:2303.16634), MT-Bench (arXiv:2306.05685), BERTScore (arXiv:1904.09675), and the BEM answer-equivalence judge of Bulian et al. (DOI 10.18653/v1/2022.emnlp-main.20), which the paper contrasts with its SES §II.B.
  • Compared against, as related work: Kamalloo et al. (arXiv:2305.06984) and Bahak et al. (arXiv:2312.07592) on LLM question-answering evaluation §II.B.
  • Superseded by: not known.

Source and license

Adapted from “SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0” by Chandra N Sekharan, Vishakan Umapathy, Abhishek Phadke (arXiv:2611.16300), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.