Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

In plain words

Doctors and nurses spend a large share of their time writing and condensing notes: radiology reports, patient messages, progress notes and recordings of visits. This paper asks whether general-purpose language models, adjusted for these jobs, can write those summaries well enough to be useful.

The authors tried eight models on four summarization tasks. They used two cheap ways of adapting a model: showing it a few example summaries in the prompt, or lightly fine-tuning it on the task's own examples. Automatic text-similarity scores do not track what a clinician cares about, so ten physicians compared the models' summaries with summaries written by medical experts, rating completeness, correctness and conciseness. For the best setup, the model's summary was judged equivalent in 45% of comparisons and better in 36%; the expert summary was better in 19%. The authors also looked at which errors could lead to harm, and at which kinds of invented content appeared.

The results come from the authors' own study. They are a reason to test such systems carefully, not a verdict that they are safe to use in clinical care.

TL;DR

• Eight LLMs (open and proprietary) are adapted to four clinical summarization tasks across six datasets, using in-context examples or QLoRA fine-tuning §3.2 tab. 1 tab. 2.
• The best setup is GPT-4 with the maximum number of in-context examples, retrieved by embedding similarity §5.1.4.
• In a ten-physician reader study on three tasks, the best model's summary was equivalent (45%) or preferred (36%) over the expert summary; the expert summary was preferred in 19% §5.2 fig. 7.
• All numbers are author-reported. The released code has a save-path bug and several differences from the paper; see Paper vs code before reusing it.

Problem

Clinicians can spend up to two hours on documentation for each hour of patient contact, and nurses up to 60% of their time on documentation §1. Summaries written by people are error-prone, and errors in medicine matter §1. Earlier work adapted models to single clinical tasks. Adapting LLMs to a diverse set of clinical summarization tasks, and showing they reach non-inferiority to medical experts, had not been done §1. General NLP benchmarks do not evaluate these clinical tasks §1.

Method

The study has three parts: model adaptation and automatic scoring on six datasets §3 §4.1; a reader study on three tasks §4.2; and a safety analysis of disparities between model and expert summaries §4.3. Figure 1 gives the overview of the evaluation framework fig. 1.

Models. Eight models in three groups §3.1 tab. 1. Open-source sequence-to-sequence models: FLAN-T5 (2.7B) and FLAN-UL2 (20B). Open-source autoregressive models: Alpaca, Med-Alpaca, Vicuna and Llama-2 (7B and 13B). Proprietary autoregressive models: GPT-3.5 and GPT-4. Alpaca and Med-Alpaca were dropped from later analysis after they performed worse than Alpaca on these tasks §5.1.1 fig. 3.

Adaptation methods. Three conditions §3.2.

1. Zero-shot prompting, with no examples (m = 0).
2. In-context learning (ICL): the m training examples nearest to the test input in an embedding space are placed in the prompt. The number m is 2^x for an integer x, chosen per model and dataset so that at most 1% of the 250 test samples are excluded for exceeding the context length §3.2.
3. QLoRA: low-rank adapters are trained on a frozen base model whose weights are stored in 4-bit precision. Used for all open-source models. Proprietary models can only use ICL §3.2.

Prompt. Each prompt has a task instruction, the example pairs, and the test input, ending with an open label for the model to fill in §4.1.1 summ_dataset.py. Generated text is scored against the expert reference summary. Temperature was set to 0.1 after a small sweep on GPT-3.5 for the Open-i dataset, and the authors state it was used for all models §4.1.1 fig. 2.

# In-context example selection and prompt  [§3.2] [§4.1.1]
# Input:  test_input (string), pool (train inputs and targets),
#         m (number of examples, m = 2^x)
# Output: prompt (string)
def build_icl_prompt(test_input, pool, m, instruction):
    # Embed the test input with the sentence encoder
    # [§3.2] [code: src/process.py#L62-L67]
    query_vector = encode(test_input)
    # Take the m pool inputs nearest to the query, nearest first
    # [§3.2] [code: src/process.py#L70-L73]
    neighbours = nearest_neighbours(query_vector, pool.inputs, m)
    # The task instruction comes first, then a blank line
    # [code: src/summ_dataset.py#L128-L130]
    prompt = instruction + "\n\n"
    # Each example is "<prefix> i: input", "<suffix> i: target", then "##"
    # [code: src/process.py#L80-L83]
    for i, example in enumerate(neighbours, start=1):
        prompt += format_example(i, example.input, example.target)
    # The test input is appended with an open suffix label
    # [§4.1.1] [code: src/summ_dataset.py#L114-L120]
    prompt += format_query(m + 1, test_input)
    return prompt

Fine-tuning (QLoRA). Settings in the Hyperparameters table below §4.1.2 train.py.

# QLoRA fine-tuning of an open-source model  [§3.2] [§4.1.2]
# Input:  base_model, train_pairs, val_pairs
# Output: adapter checkpoint for each epoch
def finetune_qlora(base_model, train_pairs, val_pairs):
    # Load the frozen base in 4-bit NF4 with double quantization
    # [code: src/train.py#L201-L206]
    model = quantize_4bit(base_model)
    # Add trainable low-rank matrices to the attention layers
    # [§3.2] [code: src/train.py#L274-L276]
    model = add_lora(model, rank=8, alpha=32, dropout=0.1)
    # Adam with weight decay, lr ramps to 1e-3 over 100 steps
    # [§4.1.2] [code: src/train.py#L151] [code: src/constants.py#L117-L119]
    optimizer, scheduler = make_optimizer(model, lr=1e-3, warmup_steps=100)
    # Train for five epochs, one optimizer step per four micro-batches
    # of size six (effective batch 24)
    # [§4.1.2] [code: src/constants.py#L115-L120]
    for epoch in range(5):
        for step, batch in enumerate(train_pairs, start=1):
            loss = model(batch).loss / 4
            loss.backward()
            if step % 4 == 0:
                optimizer.step()
                optimizer.zero_grad()
            scheduler.step()
        # Save the adapters after every epoch [code: src/train.py#L105-L107]
        save_adapter(model, epoch)
        # The authors' code computes this loss on the training split
        # (see Paper vs code) [code: src/train.py#L34]
        val_loss = mean_loss(model, val_pairs)
    return model

Evaluation metrics. BLEU, ROUGE-L, BERTScore (F1) and MEDCON, the last computed on a set of UMLS concept types §4.1.3. All are scaled to 0–100 calc_metrics.py.

Reader study. Ten board-certified physicians: five radiologists judged radiology reports, and five hospitalists judged patient questions and progress notes. Each physician rated the same 100 randomly selected inputs per task, as blinded A/B pairs of best-model and expert summaries, on a five-point scale for completeness, correctness and conciseness §4.2. Dialogue was excluded because it was too long to read §4.2 fig. A6.

Safety analysis. For radiology reports (27 pairs) and progress notes (44 pairs) where the two summaries differed in completeness or correctness, readers rated the extent and likelihood of harm if the worse summary had been used §4.3 fig. 7d.

Hyperparameters

Results

Author-reported. Setup: eight models tab. 1; six open datasets tab. 2; quantitative evaluation on 250 test samples per dataset (100 for dialogue) §4.1.2; ten physicians in the reader study §4.2. Numbers below are as printed in the paper.

Reader study, pooled over tasks (best model vs medical expert). Each comparison is rated on a five-point scale and converted to a score from −10 to 10 (positive = best model preferred). Asterisks mark p < 0.001 tab. A1.

Safety analysis (radiology and progress notes, all samples).

Fabricated information (share of samples).

Summary length (tokens, mean ± SD). Radiology reports: 47 ± 24 (model) vs 44 ± 22 (expert). Patient questions: 15 ± 5 vs 14 ± 4. Progress notes: 29 ± 7 vs 27 ± 13. Length differences were not significant (all p > 0.12) §5.2.1.

Automatic metrics, best model (GPT-4 with the maximum number of ICL examples) vs earlier work on each dataset. Full table in tab. A2. The paper says its focus is not state-of-the-art automatic scores, and that these scores should be read next to the reader study, not in place of it tab. A2.

Adaptation and model comparisons.

• Med-Alpaca, which is Alpaca tuned on medical Q&A, performed worse than Alpaca on these tasks §5.1.1 fig. 3.
• QLoRA typically beat one-example ICL for FLAN-T5 and Llama-2, but with many examples most models passed the best QLoRA model (FLAN-T5) §5.1.2 fig. 4 fig. A1.
• FLAN-T5 (2.7B) beat FLAN-UL2 (20B) under QLoRA §5.1.2.
• Adding one example already improved MEDCON over zero-shot in almost all cases; GPT-3.5 and GPT-4 led when given enough examples §5.1.3 fig. 5 fig. A1.
• Seq2seq models scored well on BLEU but worse on semantic and conceptual metrics. GPT-4 generally achieved the best performance §5.1.4 fig. 6.
• Llama-2 13B gave only marginal gains over 7B fig. A3.

Paper vs code

Code: official repository linked from the paper's Data and Code Availability statement, github.com/StanfordMIMI/clin-summ, cloned at commit 16648c7759d9380220b3cac0dd46c7f807d3e782. The LICENSE file is MIT (copyright Dave Van Veen), except src/UMLSScorer.py, which is under the UMLS license. We do not quote or reuse that file. Relevant files: src/train.py, src/run.py, src/summ_dataset.py, src/process.py, src/calc_metrics.py, src/constants.py, api/run_expmt.py.

Discrepancies, each checked against both the paper and the code:

1. Learning rate schedule. Paper: 1e-3 decays linearly to 1e-4 after 100 warm-up steps §4.1.2. Code: default is linear_decay with get_linear_schedule_with_warmup, which decays toward 0 over the run train.py constants.py. The polynomial branch that takes an end value (lrn) exists train.py, but no case in the code selects it constants.py.
2. Validation data. Paper: a separate validation set of 250 samples is used for fine-tuned models §4.1.2. Code: the validation loader is built from the training split, SummDataset(args, task='trn') train.py. Early-stopping decisions therefore use training data.
3. Early stopping. Not in the paper. The code stops when validation loss has not improved for more than PATIENCE = 5 epochs train.py constants.py. With five epochs the stop cannot fire: patience falls to 1 after the fifth epoch train.py.
4. Checkpoint. Paper does not say which epoch is evaluated. Code saves every epoch and evaluates the highest-numbered one, not the one with the best validation loss train.py run.py.
5. Temperature. Paper: 0.1 was set for all models §4.1.1. Code: 0.1 is used for proprietary models only constants.py run_expmt.py. Open-source generation is greedy decoding (do_sample=False), so temperature does not apply there run.py run.py.
6. ICL retriever. Paper cites a PubMedBERT model §3.2. Code uses pritamdeka/PubMedBERT-mnli-snli-scinli-scitail-mednli-stsb, a PubMedBERT variant fine-tuned on sentence-pair tasks constants.py. The index is a flat L2 FAISS index built by gen_faiss_idx.py gen_faiss_idx.py. That script has the dataset hard-coded to chq gen_faiss_idx.py, so other datasets need an edit first.
7. Choosing m. Paper: m = 2^x, with M set so that at most 1% of samples are excluded §3.2. Code: no such search. It uses fixed grids of example counts constants.py constants.py and hard-coded caps per dataset constants.py. Prompts that exceed the length limit are dropped summ_dataset.py.
8. GPT-4 context. Paper: 32,768 tokens tab. 1. Code: the default GPT-4 limit is 8,192 tokens; the 32k deployment is used only for the larger example caps constants.py run_expmt.py.
9. Dialogue sample size. Paper: s = 100 §4.1.2. Script default is n_samples=250 main.sh. Because the dialogue pool has 126 samples and downsampling only happens when the request is smaller than the pool, all 126 are used unless the value is changed summ_dataset.py.
10. Llama-2 weights. Paper: Llama-2 7B and 13B tab. 1. Code loads TheBloke's GPTQ 4-bit checkpoints through auto_gptq constants.py train.py. The paper does not say this.
11. FLAN-UL2 precision. Paper: QLoRA for all training §3.2. Code loads FLAN-UL2 in 8-bit instead of the 4-bit configuration train.py.
12. Token counts. The code counts tokens with tiktoken's gpt-3.5-turbo encoding for every model summ_dataset.py, which is used for the length filter and the maximum generated length.
13. Post-processing before scoring. Paper: post-processing is described only for the reader study §4.2. Code: all outputs are post-processed before scoring. It removes bracketed text and underscores and collapses duplicate adjacent words for open-source models process.py. For proprietary models, only radiology and problem-list outputs are rewritten process.py. Prompt text is stripped from GPT outputs summ_dataset.py.
14. ICL instruction. Paper: not stated. Code appends ", using the provided examples to guide word choice." to the instruction when examples are used constants.py parser.py.
15. Scores. BLEU is computed per sample and then averaged, not as a corpus score calc_metrics.py calc_metrics.py. Paper does not say which.
16. Code runs as shipped? Two defects found by reading the code (not executed):
    • src/run.py#L49 passes stopping_criteria=stopping_criteria, but that name is never defined in the file. The GPT-style path (arch gpt / gptq) would raise a NameError.
    • src/summ_dataset.py#L301 calls process.write_list_to_jsonl(path, self.data, key=None), but the function is defined as write_list_to_jsonl(filename, data_list, action='w') with no key parameter process.py. Every save would raise a TypeError. Both run.py#L81 and calc_metrics.py#L61 call this path.
17. Config placeholders. DIR_PROJECT is a placeholder and must exist constants.py. API keys and Azure resource names are None constants.py. run.py hard-codes eos_token_id = 1 for every model run.py; its effect on decoding is not checked.

The paper and the code agree on the core QLoRA values: effective batch 24 (6 × 4), five epochs, 100 warm-up steps and an initial learning rate of 1e-3 constants.py.

Pitfalls

• Reader study statistics are inconsistent in the paper. Methods say a Wilcoxon signed-rank test with Bonferroni correction §4.2. The Table A1 caption says a one-sided test tab. A1. Figure 7 gives p < 0.001 without specifying the sidedness fig. 7. Use the sidedness the implementation uses.
• Denominators are not stated for several percentages. The safety and fabrication shares are "of all samples" §5.3 or "of samples" §5.2.2; the paper does not give the count. The Figure 7c caption gives 1,500 reports per plot. This matches the design: five radiologists rate 100 radiology inputs each, and five hospitalists rate 100 inputs for each of two tasks (our arithmetic, not stated in the paper). In the safety analysis, pairs with similar completeness and correctness are counted as zero harm §5.3.
• Small reader pool and task coverage. Ten physicians across three tasks, with dialogue excluded §4.2. Progress notes come from a single ICU medical centre, which limits how far the results transfer §5.5.
• Reference quality. The paper notes that expert reference summaries can contain errors, so NLP metrics computed against them are limited §5.4. Correlations between NLP metrics and reader scores are weak, around 0.2 in magnitude §5.4 fig. 9.
• Proprietary model snapshots. The paper's internal cutoff date was 31 July 2023 §3.2. The model table notes that GPT-4's context has since been extended to 128,000 tokens tab. 1. The model snapshot used is not stated.
• Early stopping cannot fire under the default schedule. With five epochs and patience 5, the stop branch is never reached (Paper vs code, item 3). So the training length is fixed at five epochs, and the checkpoint used is the last one.
• The best setup uses many examples. Retrieval adds a per-query cost, and the maximum number of examples depends on the context limit §5.1.4.
• Context-length risk for real data. Long documents and multi-document inputs are outside the study's scope §5.5. Implementations should measure token lengths with the target model's tokenizer, since the code uses tiktoken (Paper vs code, item 12).
• Data access. Three datasets (MIMIC-CXR, MIMIC-III, ProbSum) require PhysioNet access §5.5. Data leakage into proprietary model training cannot be checked §5.5.
• Bias was not analysed. Datasets have no demographic fields, so group effects were not studied §5.5.
• Prompt and temperature were barely tuned. Each prompt was tried in one or two variants on a small dataset; temperature was tried at three values §5.5.

Lineage

• Builds on: LoRA for low-rank adaptation and QLoRA for 4-bit quantized fine-tuning, both cited as the adaptation methods used §3.2; in-context learning with retrieved examples §3.2.
• Compared against: Alpaca and Med-Alpaca (later dropped) §5.1.1; prior dataset-specific systems ImpressionGPT, RadAdapt, Med-PaLM M, ECL, CUED and ACI-Bench's own baseline tab. A2.