Benchmarking large language models for biomedical natural language processing applications and recommendations

In plain words

Medical text is growing faster than people can read it, so software is used to pull out drug names, disease names, relations between entities, topic labels, answers to medical questions, and summaries. For years the best of that software was a smaller language model that had been trained on labelled examples for each task. Chatbot-style large language models can often do such tasks without any training, just from instructions and a few examples. This paper asks how well that works in practice.

The authors tried four models on twelve medical-text datasets covering six kinds of task. They gave the models instructions with no examples, with a few fixed examples, with a few examples chosen as the most similar ones, and in one setting they trained an open model further on each dataset. They compared all of this with the older trained models. The older trained models usually came out ahead. GPT-4 was the strongest of the prompted models, especially on medical questions, and it cost far more per run than GPT-3.5. The models also sometimes dropped part of the answer, repeated themselves, or changed the answer format, and a single example in the prompt reduced these problems. The results come from the authors' own runs.

TL;DR

• Twelve biomedical datasets, six task types, four LLMs: GPT-3.5, GPT-4, LLaMA 2 13B and PMC LLaMA 13B tab. 2 p. 14.
• Supervised domain-specific BERT/BART baselines beat zero- and few-shot LLMs on 10 of 12 datasets; their macro-average is 0.6536 against 0.4862 for the best few-shot LLM tab. 3 p. 6 p. 10.
• GPT-4 with five examples leads the prompted models; on MedQA it reaches 0.7651 accuracy against a 0.4195 baseline tab. 3 p. 6.
• Fine-tuned LLaMA 2 13B has the best macro-average of the LLM settings (0.5131) tab. 3.
• GPT-4 costs 60 to 100 times more than GPT-3.5 in this study p. 8.

Problem

Biomedical literature grows by thousands of articles a day, and the tasks that turn it into structured knowledge (entity tagging, relation extraction, topic classification, question answering, summarisation, simplification) were built around supervised models that need labelled data for each task p. 2 p. 3. Those labelled sets are small in the biomedical domain, which makes supervised fine-tuning hard to scale p. 2. The authors argue that prior LLM evaluations used mostly GPT-3 or GPT-3.5, fixed the gold standard for extraction tasks, reported mostly F1-style scores, and often did not release prompts or few-shot examples p. 3 p. 4. The paper sets out to measure LLM performance on the main BioNLP task families under a common protocol, including generative tasks where the reference text is free-form p. 3 p. 4.

Method

Overview. Each model is run on the test set of each dataset under four settings: zero-shot, static few-shot (one and five examples), dynamic K-nearest few-shot (K = 1, 2, 5), and instruction fine-tuning where applicable p. 4 p. 14 p. 15. The results are compared with the reported SOTA fine-tuned BERT or BART result for the same dataset p. 14 tab. 3. Section numbers are not printed in the PDF, so citations below use page numbers.

1. Tasks and datasets. Twelve test sets across six applications: named entity recognition (BC5CDR-chemical, NCBI Disease), relation extraction (ChemProt, DDI2013), multi-label document classification (HoC, LitCovid), question answering (MedQA 5-option, PubMedQA), text summarisation (PubMed, MS^2), and text simplification (Cochrane PLS, PLOS) p. 12 tab. 2. Official metrics come from the dataset papers. Some datasets use a second metric as well p. 12.

tab. 2 lists the splits and metrics:

Source: tab. 2. For MS^2 the test gold standard is not public, so the validation set was used instead (footnote 2 of tab. 2). For PubMed summarisation, the authors dropped training and validation instances with fewer than 50 words and left the test set untouched (footnote 1 of tab. 2).

2. Baselines. The baseline for each dataset is the best reported pre-LLM result, taken from the literature and not re-run: domain-specific BERT variants (PubMedBERT, BioBERT, BioLinkBERT) for extraction, classification and QA, and BART for summarisation and simplification p. 14 tab. 3. The MedQA baseline was reported on the 4-option version; the authors ran the released model on the 5-option version for the comparison tab. 3 (footnote 1).

3. Models. GPT-3.5 and GPT-4 are used through the API as snapshots gpt-3.5-turbo-16k-0613 and gpt-4-0613 for extractive and classification tasks, and gpt-4-32k-0613 for generative tasks p. 14. LLaMA 2 13B is used as the open general-domain model. PMC LLaMA 13B is a biomedical continued pre-training of LLaMA p. 14. The paper does not say whether the LLaMA 2 model is the base or the chat variant (see Pitfalls).

4. Prompt template. One template is used for all tasks. It has five parts: task description, input specification, output specification, task guidance (for example, relation type definitions), and optional demonstrations p. 14. The prompt texts are published in the repository, one set per dataset folder, with zero-shot, one-shot and five-shot files p. 14 README.md.

5. Zero-shot and static few-shot. Zero-shot uses the template only. For static one-shot and five-shot, the authors picked training examples at random once and used the same examples for every test instance. They also published the selected examples p. 15. The static one-shot baseline is drawn as a dashed line in fig. 1 p. 28.

6. Dynamic K-nearest few-shot. For each test instance, the K most similar training instances become the examples. The method follows MedPrompt [27] p. 15. Each instance is embedded with text-embedding-ada-002, and similarity is cosine p. 15. K is 1, 2 or 5 p. 15. The repository ships precomputed ranked lists per dataset, for example similarity.rank.json.

# Dynamic K-nearest few-shot prompt  [p. 15]
# Input:  test_text; train_set (text, gold); K in {1, 2, 5}
# Output: prompt string

def build_dynamic_prompt(test_text, train_set, K):
    # Embed the test instance and all training instances with
    # text-embedding-ada-002  [p. 15]
    test_vector = embed(test_text)                    # (d,)
    # Rank training instances by cosine similarity, most similar first
    # [p. 15]; the repo reads precomputed rankings
    # [code: GPT/extractive_tasks/run_gpt.py#L260-L262]
    ranked = sort_by_cosine(test_vector, train_set)
    # Put the K nearest training examples, each with its gold output,
    # before the test instance  [p. 15]
    examples = ""
    for neighbor in ranked[:K]:
        examples += format_example(neighbor.text, neighbor.gold)
    # Task instructions, then examples, then the test input  [p. 14]
    return task_instructions + examples + format_query(test_text)

7. Decoding. Temperature is 0 for zero-shot, few-shot and GPT runs p. 15. The repo's GPT scripts use temperature 0 as well [code: GPT/extractive_tasks/run_gpt.py#L278]. The maximum generated length for LLaMA is not stated in the paper body.

8. Instruction fine-tuning. Only LLaMA 2 13B and PMC LLaMA 13B are fine-tuned, each separately on each dataset p. 14. The method is LoRA [78] with an objective that maximises the log-likelihood of the response given the instruction p. 15. Learning rate 1e-5, weight decay 1e-5, warmup ratio 0.01, three epochs p. 15.

# Instruction fine-tuning with LoRA  [p. 15]
# Input:  (instruction x, response y) pairs from one dataset
# Output: LoRA adapter weights; the base model stays frozen

def fine_tune(pairs, epochs):
    # Three epochs over the training pairs  [p. 15]
    for epoch in range(epochs):
        for x, y in pairs:
            # Maximise the log-likelihood of y given x  [p. 15]
            loss = -log_likelihood(y, given=x, params=adapter_weights)
            # The paper does not name the optimiser or batch size; the repo
            # uses paged AdamW, cosine decay, batch 1 per device
            # [code: llama/src/RE/multiple_GPU_train_V2-RE.py#L87-L107]
            adapter_weights = optimizer_step(adapter_weights, loss)
    return adapter_weights

9. Output parsing. Predictions for extraction and classification tasks are pulled from the raw output with a mix of automatic and manual steps. When a model answers in a different format (for example, a sentence with the option letter inside it), the authors extracted the answer by hand p. 15. The number of answers changed this way is not stated.

10. Statistics. Each score is bootstrapped with a subsample of 30 instances, repeated 100 times, to give a 95% interval. A two-tailed Wilcoxon rank-sum test compares settings p. 16 tab. 3 (table note).

11. Qualitative checks. Raw outputs of three LLMs under zero-shot and one-shot settings on ChemProt, HoC and MedQA were categorised as inconsistent (different format for similar inputs), missing (no answer), or hallucinated (prompt not addressed, or repetition or misinformation in the output) p. 16 tab. 4 fig. 2. For NCBI Disease the entity predictions were sorted into correct, wrong, missing and boundary-only errors p. 8 fig. 2. For PubMed summarisation, two annotators scored 50 random outputs from four systems on accuracy, completeness and readability (1 to 5), with names and order hidden p. 16 tab. 5 fig. 3.

12. Cost. Input and output tokens are counted with the OpenAI tokenizer and multiplied by the list prices on Azure's pricing page p. 16. The page date is not stated. Costs are reported per 100 instances fig. 1.

# Entity-level exact-match F1  [code: run_eval_ner.py#L4-L39, #L96-L139]
# Input:  gold_spans, pred_spans: one list of (start, end) per sentence
# Output: precision, recall, F1

def exact_match_f1(gold_spans, pred_spans):
    # Count only spans whose start and end both match
    # [code: run_eval_ner.py#L21-L26]
    true_positives = 0
    predicted_total = 0
    gold_total = 0
    for gold, pred in zip(gold_spans, pred_spans):
        true_positives += count_exact_matches(gold, pred)
        predicted_total += len(pred)
        gold_total += len(gold)
    # The repo returns 0 when a denominator is 0
    # [code: run_eval_ner.py#L42-L55]
    precision = true_positives / predicted_total
    recall = true_positives / gold_total
    # Harmonic mean of precision and recall  [code: run_eval_ner.py#L136-L139]
    return precision, recall, 2 * precision * recall / (precision + recall)

Hyperparameters

Results

Author-reported. Setup: 12 datasets, zero-shot, static one- and five-shot, dynamic K-nearest few-shot (in fig. 1), and LoRA fine-tuning. Baselines are pre-LLM fine-tuned models reported in the literature tab. 3.

Table 3 gives the primary metric per dataset. Asterisks follow the paper: an asterisk marks a best result that is significant at P < 0.05 against the other models in its group (zero/few-shot or fine-tuned) tab. 3. The paper also marks the best result per group in bold; the repository README does the same README.md.

Source: tab. 3. Column "FT" = fine-tuned. The LLaMA 2 13B five-shot column is truncated to 4,096 tokens for QA, summarisation and simplification (footnote 2 of tab. 3). The 0-shot PMC LLaMA column is not in the table because the paper fine-tunes PMC LLaMA only. The macro-average mixes metrics, which the paper flags as a limitation p. 12.

Headline findings (author-reported):

• Baselines lead. The fine-tuned baseline beats the best zero- or few-shot LLM on 10 of 12 datasets p. 6 p. 10.
• Information extraction is the biggest gap. On NCBI Disease the baseline reaches 0.9090 entity F1, against 0.6389 for the best prompted GPT-4 (five-shot) p. 8 tab. 3.
• Medical QA is the exception. GPT-4 zero-shot reaches 0.7156 on MedQA against a 0.4195 baseline. On PubMedQA, the five-shot GPT-4 result is 0.7580 against 0.7340 p. 6 tab. 3.
• Open models need fine-tuning. Zero-shot LLaMA 2 13B has a macro-average of 0.2362. Fine-tuning brings it to 0.5131 p. 7 tab. 3.
• PMC LLaMA does not beat LLaMA 2. Under fine-tuning, LLaMA 2 13B beats PMC LLaMA 13B on 10 of 12 datasets. Directly fine-tuning LLaMA 2 13B on PubMedQA gives 0.8040, against 0.7680 for PMC LLaMA on the same data p. 7 p. 11.
• Dynamic examples help the classification tasks. On LitCovid, GPT-4 macro-F1 rises from 0.5901 (static one-shot) to 0.6500 (dynamic one-nearest) and 0.7055 (dynamic five-nearest) p. 7. On PubMedQA, GPT-4 accuracy is 0.71 static one-shot, against 0.72 and 0.75 dynamic p. 7.
• More shots do not always help. Dynamic one-shot is sometimes below static one-shot, and GPT-4 with five dynamic shots is not the best in 8 of 12 datasets p. 8.

Cost (author-reported, Azure list prices p. 16):

• GPT-4 costs 60 to 100 times more than GPT-3.5 across the tasks p. 8.
• Five-shot extraction and classification with GPT-4 costs about $2 to $10 per 100 instances. GPT-3.5 costs about $0.03 to $0.16 for the same tasks p. 8 fig. 1.
• PubMed summarisation with GPT-4 five-shot costs $84.02 per 100 instances, about $5,600 for the test set. The text gives $0.71 per 100 instances for the GPT-3.5 five-shot run p. 8.

Qualitative results (author-reported):

• Errors on NCBI Disease. GPT-4 zero-shot gives 488 correct, 227 wrong, 472 missing and 328 boundary-error entities out of 960. LLaMA 2 13B zero-shot gives 148 correct, 231 wrong, 812 missing and 74 boundary-error entities. A BioBERT model from an independent study, used because the SOTA model was not public, gives 863 correct p. 8 p. 9 fig. 2.
• Inconsistent, missing and hallucinated outputs. LLaMA 2 13B zero-shot on HoC gives 102 hallucinated (32%) and 69 inconsistent (22%) outputs. With one shot, those fall to 0 and 23. On MedQA zero-shot it gives 402 inconsistent (32%) p. 9 fig. 2.
• Summarisation human scores (50 PubMed items; 1 to 5 scale). Fine-tuned BART, GPT-4 zero-shot and GPT-3.5 zero-shot are close on accuracy. GPT-4 zero-shot is significantly more readable than BART, but less complete p. 9 p. 10 tab. 5.

Human scores (tab. 5):

The readability row does not match the text on page 10, which gives 4.66 and 4.73 for GPT-3.5 and GPT-4 (see Pitfalls).

Paper vs code

The paper cites Zenodo DOI 10.5281/zenodo.14025500 as its code and data archive p. 17. That record links to the GitHub repository BIDS-Xu-Lab/Biomedical-NLP-Benchmarks. Hua Xu, a corresponding author, runs a corresponding author from Yale's biomedical informatics department p. 1. The owner name BIDS-Xu-Lab is probably that group, an inference from the affiliation. The repo is Apache-2.0 LICENSE. This card uses the default-branch HEAD e607a812… from a shallow clone. The Zenodo record points to tag v1.0.0; the tag's commit was not separately checked.

Discrepancies found after reading the code:

1. Fine-tuning hardware and batch. The paper uses eight H100 80 GB GPUs p. 15. The shipped script sets five GPU indices (1, 3, 5, 6, 7) run-RE.sh. The training code uses batch size 1 per device with no gradient accumulation multiple_GPU_train_V2-RE.py, so the effective batch depends on the GPU count. The paper does not give a batch size.
2. LoRA configuration. The paper names LoRA p. 15 but gives no rank or target modules. The code uses rank 16, alpha 64, dropout 0.05 on seven projection layers, with 4-bit NF4 quantised base weights multiple_GPU_train_V2-RE.py. The paper does not mention quantisation.
3. Tokenizer hard-coded. The script loads the tokenizer from meta-llama/Llama-2-7b-hf regardless of the model passed in multiple_GPU_train_V2-RE.py. The script also uses a variable token on that line that the file never defines. As shipped it fails at that line.
4. Fine-tuning code is elsewhere. The README says the instruction-tuning code lives in a separate repository (LLMindcraft), linked to a specific commit README.md. It is unclear whether the RE script here matches that repository. Only the PMC LLaMA model is listed in the run script run-RE.sh.
5. Snapshot names. The paper names exact snapshots p. 14. The code uses Azure deployment names gpt-4, gpt-35-turbo and gpt-4-32k, with fixed API versions run_gpt.py run_gpt.py. Nothing in the repo maps these deployments to the snapshots.
6. Static vs dynamic in the shipped script. The paper describes static one- and five-shot examples chosen at random p. 15. The extractive-task script builds one- and five-shot prompts from the similarity-ranked neighbours for every dataset run_gpt.py run_gpt.py. The static prompts are the prompt_*shot.txt files README.md. The paper does not say which set produced the static numbers in tab. 3.
7. Silent fallbacks. If an instance has no ranked neighbours, the extractive script drops the examples and sends a zero-shot prompt run_gpt.py. On an exception it writes an empty answer run_gpt.py. The generative script records an empty answer on InvalidRequestError run_gpt.py. The MedPrompt script treats "filtered" errors as empty answers and stops the whole run on any other InvalidRequestError run_medprompt.py. The paper counts missing outputs as an error type p. 16, but does not say how empty answers were counted.
8. Input truncation for GPT. The generative script truncates GPT inputs to 15,000 or 31,000 tokens, minus 2,500 for one-shot run_gpt.py. The paper body mentions truncation only for LLaMA p. 7. The MedPrompt script uses 14,000 and 30,000 run_medprompt.py.
9. Broken entry points. The extractive script's main loop is missing a colon at for gpt_version in ['gpt-4', 'gpt-3.5'] run_gpt.py. The MedPrompt script imports templates_medprompt run_medprompt.py, but the repo has GPT/generative_tasks/templates.py. Both scripts read data from absolute or zhizheng_data/ paths that do not exist in the repo run_gpt.py.
10. Macro-F1 label set. The paper does not list the labels used for macro-F1 on ChemProt, DDI, HoC or LitCovid tab. 2 p. 13. The evaluator calls the scikit-learn default, which averages over every label that occurs in gold or prediction run_eval.py run_eval.py.
11. BERTScore model. The paper gives no model for BERTScore tab. 2 p. 13. The evaluator uses bert-base-multilingual-cased run_eval.py. BART score uses facebook/bart-large-cnn with a checkpoint from a Google Drive link run_eval.py README.md.
12. Prompt wrapper. The evaluator's default is no chat wrapper evaluator.py. A mellama_prompt wrapper exists but is only used if passed in model_prompt.py. Which wrapper was used for the reported LLaMA numbers is not stated.
13. NER error categories. The paper's boundary, missing and wrong categories fig. 2 are not computed by the released NER script, which returns precision, recall and F1 only run_eval_ner.py. The categorisation code is not in the repo.
14. Base vs chat LLaMA. The paper says LLaMA 2 13B. The README's result file names say llama2_13b_chat README.md, and the example evaluation script uses Llama-2-7b-chat-hf run_evaluation.sh. The paper does not state the variant.
15. Temperature. The paper and the code agree on temperature 0 p. 15 run_gpt.py run_medprompt.py.

The release does not include the manual output-extraction step, the statistics code, the NER error-categorisation code, or the human-evaluation sheets (the supplementary workbook is shipped as an xlsx file).

Pitfalls

• The baseline is not re-run. The SOTA column comes from published numbers p. 14. Their splits and metrics may differ from the ones used here. The MedQA baseline is for 4 options; the LLMs were run on 5 options tab. 3 (footnote 1).
• Percentages in the text are mixed. Some differences are absolute points and some are relative, without saying which. "Approximately 7%" for GPT-4 vs GPT-3.5 zero-shot macro-average is 0.4561 − 0.3814 = 7.5 points, about 20% relative p. 7. The "15% higher" claim for the baselines is also absolute p. 4. Use the table values.
• Readability numbers disagree. The text on page 10 gives readability of 4.66 for GPT-3.5 and 4.73 for GPT-4, both above BART p. 10. As read from the table image, tab. 5 gives 3.57 for GPT-3.5 and 4.05 for BART. Check against the supplement before using these values.
• Hallucination count disagrees with the figure. The text gives 506 hallucinated ChemProt outputs for LLaMA 2 13B zero-shot p. 9. The bar in fig. 2 (B) appears to read 576. Both are about 3% of 16,943.
• Test-set labels disagree. tab. 2 lists 1,273 MedQA and 500 PubMedQA test items. fig. 1 labels PubMedQA as 1,273 and MedQA as 500, and fig. 2 (D) gives 1,273 for MedQA. Trust the table.
• GPT-3 vs GPT-3.5. The summarisation cost paragraph calls the $0.71 figure "GPT-3". Given the surrounding context, this is GPT-3.5 p. 8.
• Inconsistent outputs are not all errors. The authors note that inconsistent answers may still address the prompt, just in another format p. 9. Automatic extraction misses such answers, so reported accuracy depends on manual cleaning. How many outputs were changed by hand is not stated p. 15.
• Output formats are fragile. Adding one example sharply cut hallucinated and inconsistent outputs for LLaMA 2 13B p. 9. For accuracy, more examples do not always help: dynamic one-shot is sometimes below static one-shot, and dynamic five-shot is not the best GPT-4 setting in 8 of 12 datasets p. 8.
• Long inputs. LLaMA 2 13B has a 4,096-token limit. Five-shot prompts for summarisation and QA were truncated, and the authors blame the limit for the weak results there p. 7 tab. 3 (footnote 2).
• Summarisation metrics disagree with humans. Automatic metrics (ROUGE, BERTScore, BART score) favour the fine-tuned BART output. Human judges preferred GPT-4 on readability p. 11 p. 10.
• Cost is a snapshot. GPT-4 costs are based on a pricing page with no date given p. 16. The authors note newer GPT-4 versions may cost less p. 11.
• Sample sizes are small in the human study. Two annotators judged 50 items p. 16. The statistical tests are in a supplement that is not part of this card.
• Bootstrap with 30 items. Each interval uses 100 resamples of 30 instances p. 16. The intervals are wide for small datasets; the paper gives the interval values only in the supplement.
• Competing interests. Two authors have research-related financial interests at Melax Technologies p. 23.
• Macro-average mixes metrics. The paper's headline macro-average averages ROUGE-L, F1 and accuracy across tasks. The paper itself says no single metric fits all tasks p. 12.
• Fine-tuning did not help summarisation. The paper says LLaMA 2 13B fine-tuning improved all tasks except text summarisation and simplification p. 7.

Lineage

• Builds on: MedPrompt-style dynamic few-shot selection [27] (arXiv:2311.16452, p. 15). The embedding model is text-embedding-ada-002, from [77] (arXiv:2201.10005, p. 15). LoRA [78] (arXiv:2106.09685). The LLaMA family is LLaMA 2 [22] (arXiv:2307.09288). PMC LLaMA [34] is a continued pre-training on biomedical papers and textbooks p. 2.
• Compared against: SOTA domain-specific BERT and BART models from the literature (PubMedBERT [16], BioBERT [9], BioLinkBERT [95], BART [71]), not re-run tab. 3.
• Related evaluations: Table 1 compares this study with other LLM evaluations in biomedical and clinical NLP, for example [27], [38], [80]–[82] tab. 1 p. 3.