MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications

In plain words

Doctors' exam scores tell you little about whether a language model can do the job in a clinic. A model can pass medical licence-style questions and still fail at tasks such as working out a drug dose or writing a correct database query. MEDIC is a test suite built around that gap. It covers five kinds of ability and checks each one in the way that suits it: a plain right-or-wrong check where an answer can be verified, and a judge-based check where it cannot.

For written summaries and clinical notes, the paper introduces a method called the Cross-Examination Framework. It makes the text ask questions of the other text. A summary that leaves out facts, contradicts the source, or invents details fails in a measurable way, and no human-written reference is needed.

The main finding is that doing well on one kind of test does not predict doing well on another. Models that answer exam questions well can still be unreliable at calculations and queries, and models that refuse harmful requests can still miss errors in a clinical note. No single model wins everywhere, so the authors argue for choosing models per task.

TL;DR

• MEDIC is a multi-dimensional evaluation framework for clinical LLMs, with deterministic checks where possible and LLM-judged checks elsewhere §2.
• The Cross-Examination Framework (CEF) scores generated summaries and notes without reference texts, using YES/NO/IDK answers to generated questions §3.2 alg. 1.
• Author-reported: knowledge benchmarks saturate, operational tasks lag, and passive safety (refusal) does not predict active safety (error detection) §3.3 §3.4.
• The paper publishes a leaderboard and the CEF code. The MEDIC evaluation harness itself is not published as code.

Problem

Standardised medical exams have become saturated for frontier LLMs, and they do not test the operational tasks that clinical workflows need, such as calculation, structured query generation or auditing a note §1. Deployment pilots are the true test but are costly and slow, so the paper asks for upfront "leading indicators" that help pick a model before deployment §1. Clinical summaries and notes also cannot be scored well with n-gram metrics such as ROUGE and BLEU, which the authors say do not verify facts §2.1 app. CEF. The paper assumes that the tasks can be grouped by how deterministic they are, and that open-ended outputs need a proxy judge §2.

Method

Dimensions and task mapping

MEDIC organises tasks under five dimensions: Medical reasoning, Ethics and bias, Data and language understanding, In-context learning and Clinical safety §2. Each task has a primary or secondary dimension. The mapping is given in tab. 1. The paper also states that this taxonomy was not validated by a formal clinician panel, unlike MedHELM, though a practising clinician helped build the dimensions §2.1.

Task families and their checks, as described in §2.1 and [app. evaluation tasks]:

1. Knowledge retrieval and reasoning. MedQA (USMLE style), MedMCQA, MMLU-Pro and PubMedQA are scored by exact match of the chosen option. GSM8K and AIME ('24 and '25) act as general-reasoning controls against overfitting. The appendix also says standard MMLU is employed, but MMLU is not listed in tab. 1 or in the resource list app. resources.
2. Clinical generation and verification. Discharge summaries (DischargeMe), summarisation of clinical trial protocols and internal-medicine problem lists, and note generation from dialogue (ACI-Bench and SOAP notes) are scored by the CEF §2.1 app. CEF.
3. Functional execution. EHRSQL checks executable SQL and abstention on unanswerable questions app. eval tasks. MedCalc checks a clinical number from a patient note app. eval tasks. IFEval checks strict instruction following app. eval tasks.
4. Open-ended inquiry. Pairwise comparison by an LLM judge over MedicationQA, HealthSearchQA and ExpertQA, aggregated into Elo ratings §2.1 [app. eval tasks]. HealthBench (rubric score) appears in tab. 1 and in the results overview app. results.
5. Safety and audit. Med-Safety scores harmfulness from 1 (full refusal) to 5 (full compliance) with an LLM judge app. eval tasks. ToxiGen is listed in tab. 1. MEDEC tests error flagging, sentence localisation and correction §2.1 app. eval tasks.

Evaluation is hybrid. Structured tasks use objective metrics. Open-ended tasks use an LLM as judge, since reference-based metrics are weak for them §2.

Cross-Examination Framework (CEF)

CEF is a reference-free method for generated documents, such as summaries and notes app. CEF. It has three steps alg. 1:

1. Generate N closed questions from the source document D, and N' from the generated text S. Each question must be answerable with YES only, so every ground-truth answer is YES. Questions are derived from the text they test app. CEF.
2. Cross-examine. Answer the summary-derived questions using only D, and the document-derived questions using only S. Each answer is one of YES, NO or IDK ("I don't know") app. CEF alg. 1.
3. Score the answers. Coverage, Conformity and Consistency are percentages alg. 1. Conciseness is defined in the appendix as word-level token reduction from D to S app. CEF.

The appendix gives YES-only questions as the design choice, supported by an ablation in the authors' earlier work. That ablation reports that mixed YES/NO/IDK questions were unstable on re-evaluation, while YES-only questions kept 99.4% stability app. CEF (cited as raha2026cross in §3.2).

Judge and scoring protocol

• Open-ended QA: the judge picks the better of two answers to the same question, with both presentation orders. A win counts only when the same model wins in both orders; otherwise the result is a draw §2.1. Wins and losses are aggregated into Elo ratings §2.1 app. eval tasks.
• The pairwise judge scores the five criteria relevance, completeness, safety, ethics and clarity §3.5 (rubric in the appendix prompts section).
• Med-Safety harmfulness is scored by Llama-3.1-70B-Instruct, following AMA principles app. results.

# Cross-Examination Framework scores  [alg. 1] [app. CEF]
# Input:  document D (text), summary S (text)
# Output: coverage, conformity, consistency, conciseness

def cef_scores(document, summary):
    # Step 1: YES-only questions from each text  [app. CEF]
    doc_questions = yes_questions(document)    # Q_D, all answers YES
    sum_questions = yes_questions(summary)     # Q_S, all answers YES

    # Step 2: cross-examine, each answer in {YES, NO, IDK}  [alg. 1 l.3-4]
    doc_q_on_sum = answer_all(summary, doc_questions)
    sum_q_on_doc = answer_all(document, sum_questions)

    # Step 3: scores  [alg. 1 l.5-8]
    coverage = 100 - percent(doc_q_on_sum == "IDK")
    conformity = 100 - percent(doc_q_on_sum == "NO")   # contradictions
    consistency = 100 - percent(sum_q_on_doc == "IDK")  # hallucinations
    conciseness = token_reduction(document, summary)   # [app. CEF]
    return coverage, conformity, consistency, conciseness


def percent(flags):
    # Share of True values, in percent  [alg. 1]
    return 100 * sum(flags) / len(flags)

# Overall summarisation score  [app. eval tasks]
# Input: coverage, conformity, consistency, conciseness (each 0-100)
# Output: one number

def summarisation_overall(coverage, conformity, consistency, conciseness):
    # Harmonic mean of coverage and conciseness, 0 unless both positive
    if coverage > 0 and conciseness > 0:
        harmonic = 2 * coverage * conciseness
        harmonic = harmonic / (coverage + conciseness)
    else:
        harmonic = 0
    # Average of the three terms and the harmonic mean  [app. eval tasks]
    terms = [coverage, conformity, consistency, harmonic]
    return sum(terms) / len(terms)

# Pairwise verdict with swapped order  [§2.1]
# Input: judge(question, first, second) returns "first" or "second"
# Output: "a", "b" or "draw"

def pair_verdict(judge, question, answer_a, answer_b):
    # Present both orders to the judge  [§2.1]
    order_ab = judge(question, answer_a, answer_b)
    order_ba = judge(question, answer_b, answer_a)
    # A win counts only if it holds in both orders  [§2.1]
    if order_ab == "first" and order_ba == "second":
        return "a"
    if order_ab == "second" and order_ba == "first":
        return "b"
    return "draw"

# MedCalc answer check  [app. eval tasks]
# Input: predicted value, ground-truth value, whether the target is continuous
# Output: bool

def medcalc_correct(predicted, truth, is_continuous):
    # Continuous targets: within 5% of the ground truth  [app. eval tasks]
    if is_continuous:
        return abs(predicted - truth) <= 0.05 * abs(truth)
    # Discrete targets: exact match  [app. eval tasks]
    return predicted == truth

Hyperparameters

Decoding for the task models app. generation parameters. "Greedy" is temperature 0. MCQ tasks are scored from logits and have no generation. Top-k is not set for any task.

Judge decoding app. judge decoding parameters:

Other settings:

Results

Author-reported. Setup: models were scored under uniform settings, so the authors say scores may differ from vendor-reported numbers §3.1. The appendix overview covers 14 models; Tables 2 and 3 each cover 11 tab. 2 tab. 3. Clinician validation is limited to one clinician app. clinical validation.

Functional and knowledge tasks tab. 2. Accuracy in percent for MedQA and MedMCQA and MedCalc; EHRSQL uses RS(0) tab. 2. Best per column in the source table: MedQA Kimi-K2-Thinking, MedMCQA Llama-4-Maverick, MedCalc Mistral-Large-3-675B, EHRSQL GPT-OSS-120B tab. 2.

MEDEC tab. 3. Error flag and sentence detection are accuracies in percent. Generation metrics compare the proposed correction with the reference.

Clinician validation app. clinical validation. Mean error between clinician and model-judge on a 1 to 5 scale, with 95% CI, for 73 questions. The sign convention is not stated; the text suggests positive means the judge scored higher than the clinician (inferred).

Overview across all tasks app. results. The appendix table gives a rank and score per model and task. The top-ranked model per task is listed here. Metric names are not given per row; Med Safety is lower-is-better.

Findings the paper reports.

• Knowledge tasks are near saturation: top models have median above 75%, while MedCalc and EHRSQL have median below 40% and wider spread §3.3 fig. 4a.
• Passive safety is near optimal: most models score close to 1 on Med-Safety, but MEDEC stage 1 accuracy drops sharply and several models are near zero §3.4 fig. 4b tab. 3.
• CEF: larger models tend to show lower conformity; lexical metrics (BLEU, ROUGE, BERTScore) show negligible Spearman correlation with CEF on ACI-Bench §3.2 fig. 3b, fig. 3c.
• Elo rankings are stable across three judges with Spearman rho of at least 0.98 §3.5 fig. 5b. GPT-OSS-120B is reported as consistently top-tier §3.5 fig. 5a.
• The authors' earlier CEF validation reports 78.4% alignment with semantic error annotations and 65.7% with the full FRANK taxonomy §3.2 (cited work, not re-run here).

Paper vs code

The only code the paper publishes is the CEF implementation. The repository is m42-health/cross-examination-framework, cloned at commit 9ec5bb48 on its default branch. The paper links the cef-paper/cef-paper branch path, and the default branch contains the same cef-paper/ directory; I did not compare the two. The repository's README describes the code as the official code for a separate CEF paper (arXiv:2601.19350), not for MEDIC. The MEDIC harness is not in this repository. The other tasks (MedCalc, EHRSQL, MEDEC, DischargeMe, Med-Safety, lm-eval) point to third-party repositories, and the paper also links an M42 HealthBench repository. I did not clone any of these. The repository is CC BY-NC 4.0, so this section cites code and does not quote it.

• Consistency and coverage definitions, two versions. The paper defines Consistency as 100 minus the percent of IDK answers to summary questions answered from the document alg. 1. The older evaluator in utils.py instead reports YES fractions, and it answers the summary's own questions against the summary utils.py. The later recalculations match the paper and use 100 minus percent IDK utils.py explore_results.py. Use the recalculated definitions only.
• Conformity. Paper: 100 minus percent of questions where the document answer is YES and the summary answer is NO alg. 1. Code: 100 minus percent NO over all answers utils.py. These agree only if all document answers are YES, which the code does not enforce (see Pitfalls).
• Conciseness direction and unit. The appendix defines conciseness as a token reduction, so higher means shorter app. CEF. The code computes a character-length ratio, len(summary) divided by len(source), times 100 utils.py. Higher means longer, so the sign is opposite. The explore script labels this "Lower = more concise" explore_results.py.
• Conciseness for notes. The appendix says conciseness is not computed for note generation app. eval tasks. The note-generation code computes it anyway utils.py.
• Overall summarisation score. The harmonic-mean aggregate in the appendix app. eval tasks does not appear in cef-paper/src. A search for "harmonic" there returns nothing.
• Generation decoding. The paper gives temperature 0.1 and top-p 1.0 for summarisation and note generation app. generation parameters. The note-generation code uses temperature 0.0 and top-p 0.9, retrying with temperature 0.1 times the attempt number generate_notes.py generate_notes.py. The summarisation generator uses temperature 0.0 and top-p 0.9 generate_highlights.py. The question and answer calls use temperature 0 and top-p 0.9 [code: cef-paper/src/summarization/utils.py#L200-L202] [code: cef-paper/src/summarization/utils.py#L218-L220]. The summary answer budget is 100 tokens, and the note answer budget is 20000 answer_generator.py.
• Datasets in the code. The summarisation code targets CNN/DailyMail generate_highlights.py, and the note code targets ACI-Bench generate_notes.py. The MEDIC clinical summarisation sets (clinical trials, problem lists) and the DischargeMe and SOAP data are not in the repository.
• Question-generation prompts. The prompts ask for YES-only closed questions and for IDK when the text does not answer prompts.yaml prompts.yaml.
• Default question counts. The summarisation evaluator defaults to 10 source questions and 5 summary questions utils.py, while the note pipeline uses 10 and 10 question_generator.py. The paper gives no N or N' in text.

Pitfalls

• The MEDIC harness is not published. Only the CEF code and the leaderboard are public. The MedCalc, EHRSQL, MEDEC, DischargeMe and Med-Safety runs are adapted from other repositories app. resources. Matching the reported numbers needs the prompts app. prompts for reproducibility and the decoding settings above. The prompt texts were not read in full for this card.
• Answer parsing is by substring. The parser checks "YES", then "NO", then "IDK" anywhere in the reply utils.py. A reply such as "IDK, the note does not say NOT" parses as NO. Constrain the output or parse the first token.
• Question parsing ignores the generator's answer. The parser keeps any item with "question" and "answer" keys and does not require YES utils.py. A question the generator labelled NO still enters the cross-examination. The paper assumes every question is YES app. CEF.
• Model sets differ across tables. Phi-4 appears only in the MEDEC table tab. 3, while Meta-Llama-3-70B, OpenBioLLM-70B and Med42-v2-70B appear only in the appendix overview app. results. Table 2 has 11 models. Do not assume a model was tested on every task.
• Units differ across tables. Table 2 reports percentages (for example MedQA 83.00) tab. 2, while the appendix reports fractions (for example 0.83) app. results. Check the unit before comparing.
• Missing values are architectural. NA cells mean the model could not run the long-context task. The paper does not give the context limit or the generation budget that caused it §3.1.
• MMLU is mentioned but absent from the task map. The appendix says standard MMLU is used app. eval tasks, while tab. 1 and the resource list do not include it.
• Clinical validation is weak evidence. It uses one clinician and 73 questions app. clinical validation. The text calls the deviations small, but Llama-3.1-70B (-0.71) and Qwen2.5-72B (-0.55) have confidence intervals that exclude zero on risk mitigation, and the text does not discuss this.
• Three judges give a thin agreement test. The rho of at least 0.98 is computed over three judge models fig. 5b. It shows the rankings agree across these judges, not that they match clinicians.
• Elo details are not stated. The Elo update rule, initial rating, K factor and any bootstrap are not stated §2.1 app. eval tasks. Rankings computed with a different update rule may differ.
• MedCalc tolerance. "Within 5%" is read here as relative error. The paper does not say how a zero ground truth is handled.
• Some Elo judges are also rated models. Two of the three Elo judges, Qwen2.5-72B-Instruct and DeepSeek-V3.1, are also in the evaluated model list fig. 5 app. resources. The paper names self-preference bias as a limitation §4 but does not measure it for these judges.
• Dimension names differ. The introduction uses "Data and language understanding" §1, and the discussion uses "Data agency" §4. Use the introduction's names.
• MEDEC coverage is partial. The UW test set was not available, so only the MEDEC-MS collection is used, and the restriction to single-error notes is a stated limitation app. eval tasks.
• Sampling is not greedy. Summarisation, note generation and open-ended QA use temperature 0.1 to 0.2 with multinomial sampling [app. generation parameters], so repeated runs can differ. Seeds and repeat runs are not stated.
• ToxiGen metric is not stated. Table 1 lists refusal rate and harm score for Med-Safety and ToxiGen tab. 1. The appendix says ToxiGen is run with lm-eval-harness app. resources, and the metric name is not given.

Lineage

• Builds on: CEF from the authors' earlier work (cited as raha2026cross in §3.2). The CEF repository's README gives arXiv:2601.19350 for it. The MEDIC paper itself relies on it for the summarisation and note-generation scores.
• Compared against: MedHELM, a clinician-validated taxonomy §2.1 §4, and HealthBench, which used physicians to write criteria §4. The paper says MEDIC is orthogonal to taxonomy breadth and focuses on the gap between recall and operational agency §4.
• Related frameworks named in the appendix: CLUE and MedHELM app. related work.
• Superseded by: not stated.