Med-HALT: Medical Domain Hallucination Test for Large Language Models

In plain words

Large language models sometimes state false facts with full confidence. In medicine that is risky, so the authors built a test that measures how often this happens. The questions come from medical licensing and entrance exams in several countries, plus a set of PubMed lookups.

Some tests give the model a multiple-choice question and ask it to judge an answer, including a suggested wrong one, an option that says "none of the above", or a made-up question. Other tests ask the model to recall a paper's title, link or PubMed ID from one of the other fields. Answers are checked against known results, and wrong answers cost points, so a model that guesses gets less credit than one that is right. The authors ran open models and two OpenAI models through the tests and report large differences between them.

TL;DR

Med-HALT is a benchmark, not a training method. It has three reasoning tests built from multiple-choice exam items and four memory tests built from PubMed records. The main score adds a point for each correct answer and takes away 0.25 for each wrong one. The authors report that Llama-2 70B is best on the false-confidence test and that Falcon 40B is best on the fake-question test and ties for the top memory-test accuracy (40.46%) on three of the four tests. The dataset is restricted to non-commercial use, and the code repository has no LICENSE file, so check both before reusing anything.

Problem

Language models produce plausible but unverified or incorrect text, and the authors argue that this is especially harmful in healthcare §1. Earlier hallucination work covers other domains, but the authors state that no such evaluation existed for the medical domain at the time §1. Med-HALT is meant to fill that gap with a fixed test set that can be run on any model §1.

The authors also note that instruction-tuned models may hallucinate more than their base versions §6.1, and that prompt wording, decoding settings and even repeated calls to the same API change the results §7.3 §7.4 §7.5.

Method

Med-HALT is a set of seven test sets, each defined by an input format and an expected output, plus one scoring rule §1.1 §3. Each item is sent to a model with a fixed instruction, a required output format and a few worked examples. The model's answer is parsed and compared with the known answer §5.2 eval_full.py.

1. Test sets

Test sets are built from five sources. Four are medical exam collections: the MedMCQA questions from two Indian postgraduate entrance exams §2, a Spanish residency exam (HEAD-QA) §2, and the USMLE and Taiwanese licensing exams §2. The fifth is PubMed records, used for the memory tests §2. Table 1 gives the question counts for the exam sources, Table 7 gives the sizes of the seven test sets, and Table 6 gives the PubMed statistics tab. 1 tab. 6 tab. 7.

2. Reasoning tests (RHT) §3.1

The reasoning tests take a multiple-choice question and a set of options. The model's output is a set of per-option decisions, and the item is scored as correct or incorrect §1.1.

• False Confidence Test (FCT). The prompt includes one randomly suggested answer. The model must say whether that answer is correct, give the answer it believes is right, and explain both the correct and the incorrect options §3.1.
• None of the Above (NOTA). The correct option is replaced by "none of the above". The model must choose an option and explain it §3.1.
• Fake Questions Test (FQT). The question is made up, either by human experts or by GPT-3.5 §3.1. The model must recognise that the question cannot be answered §3.1.

The output schema for FCT asks for a yes/no verdict, the answer, and two explanations §3.1. For NOTA and fake questions it asks for the chosen option, its index and two explanations §3.1.

3. Memory tests (MHT) §3.2

Each memory test gives one PubMed field and asks for another, and a model can answer that a paper does not exist:

• PMID to title. Input: PubMed ID. Output: title §3.2.
• Title to link. Input: title. Output: PubMed URL §3.2.
• Abstract to link. Input: abstract. Output: PubMed URL §3.2.
• Link to title. Input: PubMed URL. Output: title §3.2.

The output schema contains an existence flag and the requested field §3.2. The scorer compares only the requested field eval_full.py.

4. Prompting and inference §5.2

Prompts are built from a fixed instruction (prompt version v0), the required output format, two worked examples and the test item utils.py utils.py. Each example ends with a stop marker, and generation is stopped at that marker utils.py model.py.

# Build the prompt for one test item  [§5.2]
# Input:  instruction, output_format, shots (input, output pairs), item_text
# Output: a prompt string that ends where the model answers

def build_prompt(instruction, output_format, shots, item_text):
    # Instruction, then the required output format
    # [code: medhalt/prompts/utils.py#L149]
    header = instruction + "\n" + output_format + "\n"
    # Worked examples, each closed by the stop marker
    # [code: medhalt/prompts/utils.py#L147]
    examples = "Examples: \n"
    for shot_input, shot_output in shots:
        examples += (
            f"Input : {shot_input}\nOutput: {shot_output}Stop Here\n\n"
        )
    # The test item goes last, with an empty output slot
    # [code: medhalt/prompts/utils.py#L37-L39]
    return header + examples + item_text

Number of worked examples: the code uses two, one drawn from the default examples and one from the task-specific examples utils.py. The paper does not state the number used for its main results §5.2.

5. Scoring §5.3 eq. 1

The paper scores each item with a pointwise rule: a correct answer earns P_c = 1 and a wrong one earns P_w = -0.25. The paper defines the final score as the average over all N items §5.3 eq. 1.

S = \frac{1}{N} \sum_{i=1}^{N} \left( I(y_i = \hat{y}_i)\, P_c + I(y_i \neq \hat{y}_i)\, P_w \right) eq. 1

# Pointwise score over one test set  [eq. 1] [§5.3]
# Input:  predictions, labels (N items each)
# Output: average points per item

def pointwise_score(predictions, labels, p_correct, p_wrong):
    # Points per item: p_correct if right, p_wrong if wrong  [eq. 1]
    points = [
        p_correct if pred == label else p_wrong
        for pred, label in zip(predictions, labels)
    ]
    # Average over all N items  [eq. 1]
    return sum(points) / len(points)

The repository does not compute this average. It computes a different number, described in Paper vs code.

Hyperparameters

Results

Author-reported. Setup: Med-HALT test sets tab. 1 tab. 7; models run in their default configuration, with no fine-tuning or hyperparameter changes §5.1; the OpenAI models were accessed through the Azure API §5.2. The paper compares the models only with each other, and reports no other baseline §5.1.

Reasoning tests tab. 2

Accuracy is a percentage. "Score" is the value printed by the authors. See Pitfalls for how it is computed.

Llama-2 70B has the highest FCT accuracy, but the authors note that no model reaches an acceptable level on that test §6. Falcon 40B and Falcon 40B-instruct are highest on the fake-question test §6 tab. 2.

Memory tests tab. 3

The authors report that Falcon models beat GPT-3.5 and Text-Davinci on the memory tests §6. That statement does not hold for every test, see Pitfalls.

Prompt sensitivity and format errors

• Prompt framing (three variants of one prompt, task and model not stated): accuracies of 24.44, 22.97 and 25.48 tab. 4 §7.3.
• Repeated GPT-3.5 runs: highest accuracy 28.52% and lowest about 27.87% on repeated samples §7.4 fig. 7.
• Zero-shot GPT-3.5 accuracy: 7.31%, rising with more examples and levelling off after about three §7.2 fig. 6.
• Format errors (outputs that could not be parsed): GPT-3.5 2.24% on FCT and Llama-2 70B-chat 41.1% on FCT and 24.92% on NOTA tab. 5.
• Temperature: GPT-3.5 accuracy varies only slightly over the temperatures tried §7.1 fig. 5.

Paper vs code

Code at commit 2fed21ed696a1949b1fd6dfdbd7a08d792e0e05f README.md. Several discrepancies, each with both citations:

• Score formula. The paper defines the score as an average over N items §5.3 eq. 1. The code computes (correct × 1 + wrong × P_w) / 100, dividing by the constant 100, not by N eval_full.py. The reported Score column follows the code formula. For example, Falcon 40B on fake questions has 99.89% accuracy on 1,858 items, which gives 1,856 correct and 2 wrong. The code formula gives (1,856 − 0.5) / 100 = 18.56, matching the table tab. 2. The paper's average would be about 0.999. Scores above 100 in the NOTA column, such as 188.66 for Llama-2 70B, cannot be averages of values in [-0.25, 1].
• Decoding. The paper says open models used temperature 0.6 and top-p 0.95 §5.2. run_inference.sh passes temperature 0.6 run_inference.sh run_inference.sh. But model.py samples only when --greedy is passed, because the flag is inverted: it is stored as false and do_sample is set to its negation model.py model.py. The script does not pass the flag, so decoding is greedy and temperature and top-p have no effect. The paper does not say whether its open-model results were sampled. The model.py default temperature is 0.2 model.py, which differs from the script's 0.6.
• FCT scoring field. The FCT prompt asks for a yes/no verdict in is_answer_correct §3.1. The scorer reads the answer field instead, and compares it to the known answer eval_full.py. The verdict field is never used.
• Fake-question scoring. The paper does not state a rule for fake questions. The code counts an answer as correct if it contains any term from a fixed list (for example "none of the above", "absurd", "fiction") and never compares it to the label eval_full.py.
• Output parsing. Appendix B says the Promptify library was used [App. B]. Neither Promptify nor any other parser package appears in the repository pyproject.toml. The code keeps only the first line of the output, then applies a regular expression and ast.literal_eval, with a regex fallback evaluate.py.
• Worked examples. The paper does not state how many worked examples the main results used §5.2. The code uses two utils.py. The repository has no command-line option to change the count, so the shot-count experiment of Figure 6 has no matching code path in the repository.
• Open model list. The shipped run_eval.sh lists the ten open models in the paper's tables run_eval.sh. No code for the two OpenAI models is in the repository.

The paper does not state a matching rule for memory tests. The code uses exact, case-insensitive matching (see Pitfalls).

Pitfalls

• Score scale. Use the code's formula only if you want to match the published Score column. The paper's Equation 1 is an average and will give different numbers. Decide which one you need and say so.
• Sampling is off by default. Under run_inference.sh, decoding is greedy, so the temperature 0.6 in the paper does not change the outputs. To sample, pass --greedy (the flag is inverted, see Paper vs code). The name suggests the opposite. The run writes its generation settings to gen_kwargs.json next to the predictions, so check that file to confirm the setting.
• Unseeded few-shot choice. Worked examples are drawn with random.sample and no seed is set utils.py. The generation seed (42) does not fix this. Two runs can use different examples.
• Parse truncation. Only the first line of each output is parsed [code: evaluate.py#L41]. Multi-line JSON loses its later lines. A string replacement also turns 's into s before parsing [code: evaluate.py#L44]. Parse failures are counted as exceptions and dropped from the accuracy denominator eval_full.py.
• Silent loss on server errors. When a request to the inference server fails, the code writes an error id for that batch model.py. The merge with the test set is an inner join on id evaluate.py, so those rows vanish and the totals shrink without warning.
• Exact string matching. Titles and URLs must match the answer key exactly after lowercasing eval_full.py. Punctuation, spacing or an extra suffix counts as wrong. The code has no special case for an "Unknown" answer, even though the paper's examples show "Unknown" as the expected answer for made-up papers [App. B, abstract-to-link example].
• Keyword scoring hides real errors. A fake-question answer counts as correct if it contains almost any negative phrase, whether or not the question is actually fake eval_full.py. Treat the fake-question accuracy as an upper bound.
• Text and table disagree. Section 2 gives the USMLE count as 2,801 and the Taiwan count as 2,482 §2. Table 1 gives the reverse (USMLE 2,482, Taiwan 2,801) tab. 1. The table's row sum, 18,866, matches the reasoning test size. The text's Falcon 40B average (42.46%) does not match Table 3 (30.36%) §6. The text gives 18.56 for Falcon 40B-instruct on fake questions, but Table 2 shows 18.43 §6 tab. 2.
• Headline claim. The paper says the open models beat the OpenAI models on all hallucination tasks §6. Table 2 shows GPT-3.5 ahead of Falcon 40B on FCT (34.15 vs 18.66) tab. 2. Table 3 shows GPT-3.5 ahead of Llama-2 70B on title-to-link (39.10 vs 14.79) tab. 3.
• Repeated values. Several memory-test cells show exactly 40.46 accuracy, for example Falcon 40B on three tests and MPT 7B on one tab. 3. The paper does not explain these values. Check them before quoting the table.
• Closed-model results cannot be reproduced from the repository. The repository has no OpenAI client code. The paper says that OpenAI output cannot be made repeatable §7.4, and the model snapshot dates are not stated.
• Dataset licence. The project page states that the Hugging Face dataset is restricted to research and non-commercial use (checked 2026-10-18). The paper's CC BY 4.0 licence covers the paper, not the dataset. This card uses no dataset content. Do not copy example items or the dataset files into anything commercial.
• Code licence unverified. The repository has no LICENSE file, and GitHub's license API returns no licence. The project page says Apache-2.0, which is not confirmed by the repository. Do not copy code verbatim until the licence is confirmed.
• Table numbering. Table and figure numbers follow the order of captions in the LaTeX source. The compiled paper may number them differently.

Lineage

• Builds on: the source exam collections (MedMCQA, HEAD-QA, MedQA) §2. These have no cards here, and the paper names no earlier method it extends.
• Compared against: Text-Davinci and GPT-3.5 (OpenAI) and Llama-2, Falcon and MPT, in default configuration §5.1 tab. 2 tab. 3.
• Superseded by: not stated in the wiki.