LLMs Accelerate Annotation for Medical Information Extraction

In plain words

Doctors' notes contain useful facts about medicines, such as the drug name, the dose and how often it is taken, but the facts sit in free text. To train a computer to find them, someone first has to mark them by hand in many notes. That work is slow and needs medical training.

The authors let a large language model make the first pass. They show it a few worked examples of the task, then give it one piece of a note at a time. The model's output is turned into structured labels, and then an expert checks and corrects them. The authors compare this with the usual route, where a person makes the first pass and an expert then corrects it. The label quality ends up close, and experts spend much less time per note.

The model was not trained or tuned for this task. The authors describe the work as a proof of concept that relies only on careful prompt writing.

TL;DR

• Two-stage pipeline: a few-shot prompted LLM writes base labels, then expert raters refine them; the comparison is against human base labels followed by the same expert refinement §4 §5.
• Two prompt schemas (token-level IOB tags and a grouped YAML output) are run separately and merged by an ensemble, which was chosen on the development set §5.3 §5.4 app. A.4.
• Author-reported on 216 test documents: refinement time drops from 26.67 to 11.32 minutes per document (-58%), and phrase-level vertical F1 is 0.907 against 0.910 for the human base tab. 2.
• The LLM path has no human base cost, so the headline saving does not count LLM compute time, which the authors say cannot be compared directly app. B.

Problem

Training medical NLP models needs many labelled examples, and expert annotation is slow and expensive. Clinical notes need multiple annotators or several rounds to reach a reliable set of labelled spans §1. The authors aim to cut the human base-labelling load while keeping expert-level label quality §1 §4.

Large language models can produce labels without training, but their outputs are unconstrained and may miss the expert standard §1. The method keeps an expert in the loop, so the LLM output is a draft that experts refine §4 §5 step 5.

Method

Task and data

The data is the 2009 i2b2 medication dataset: de-identified discharge summaries, labelled for medication fields and for which fields belong together in one medication entry §3.1 §3.2. The reason field is excluded, and the narrative field is not used §3.1 footnote. The 261 documents are split randomly into a pilot set of 16 documents (used to train raters and pick experts), a development set of 29 documents (used to choose LLM settings) and a test set of 216 documents (used for all reported results) §3.3 tab. 1.

Human annotation

Seven raters with medical backgrounds annotate. A pilot picks three of them as expert raters §4. Two phases follow. In the base phase, a rater labels a blank document §4. In the refinement phase, an expert corrects a base set produced by someone else, either a human rater or the LLM pipeline §4. Refinements are done only by expert raters §4 §6.3.

LLM labelling pipeline

The pipeline has three parts: the model, a task prompt with an input chunk, and a resolver module that turns generated text into structured entries §5. The model is PaLM 2, which the authors used without tuning for this task §5.

Steps for each document §5 fig. 3:

1. Split the document into chunks at line breaks and sentence ends §5 step 1.
2. Place each chunk in a few-shot prompt for medication extraction §5 step 2 §5.2 fig. 4.
3. Run the LLM on the prompt and keep its text output §5 step 3.
4. Resolve the text into structured medication entries §5 step 4 app. A.1.
5. An optional human refinement follows §5 step 5.

Prompt schemas §5.3:

• IOB-Token: each token gets an IOB tag from a field-type list plus a group tag that links tokens into one medication entry fig. 5.
• Direct Chunk: the model writes the field values and their group for a whole chunk, with character positions, in YAML fig. 6 app. F.2.

Resolution app. A.1: IOB output is parsed one line per token with comma-delimited fields. YAML output is read from a fenced block with a standard YAML parser. Unparsable output is logged and not used. Generated text is matched back to the source text with fuzzy matching, which tolerates small character differences. Output that cannot be matched is also logged and dropped.

Prompt selection and ensemble

The authors wrote 10 versions of the IOB prompt and 5 versions of the Direct Chunk prompt, improving them with error analysis on the development set §5.4 app. A.2. They ran a partial grid search over chunk size, decoding steps and temperature app. A.3. They used F2 as the selection score, which weights recall higher than precision §5.4 app. A.3. The final LLM base is an ensemble of the best IOB and Direct Chunk configurations §5.4 app. A.5 tab. 5. The ensemble merges the two annotation sets: entries are aligned by overlap of the medication-name span, and fields are combined as described in the pseudocode below app. A.4 fig. 9.

# Ensemble of IOB-Token and Direct Chunk annotations  [fig. 9] [app. A.4]
# Input:  model1_entries, model2_entries (same document)
# Output: merged list of medication entries

def ensemble_document_annotations(model1_entries, model2_entries):
    # Pair entries whose medication-name spans partly overlap  [fig. 9]
    aligned = align_by_name_span(model1_entries, model2_entries)
    merged = []
    for pair in aligned:
        new_entry = []
        for field in pair.fields:
            # Only model 1 has this field: keep it  [fig. 9]
            if field.model1 and not field.model2:
                new_entry.append(field.model1)
            # Both models have it and the spans overlap: keep the union
            # of the two spans  [fig. 9]
            elif field.model1 and overlaps(field.model1, field.model2):
                new_entry.append(union(field.model1, field.model2))
            # Both models have it, no overlap: keep model 1 only  [fig. 9]
            elif field.model1:
                new_entry.append(field.model1)
            # Only model 2 has this field: keep it  [fig. 9]
            elif field.model2:
                new_entry.append(field.model2)
        if new_entry:
            merged.append(new_entry)
    return merged

Fixed settings §5.4 app. A.3 app. B: temperature 0.0 (deterministic); maximum decoding steps 1024; chunk size 250 for IOB-Token and 180 for Direct Chunk. The text does not state the chunk-size unit; the appendix says characters app. A.3.

Evaluation metrics

Quality is measured at phrase level with two metrics from the i2b2 challenge §6.2. Both use precision, recall and F1, with counts summed over all documents before the ratios are taken fig. 10 fig. 11 app. C.

• Vertical: a field is a tuple of field type and character span. The model's field set is compared with the ground-truth set in each document fig. 10.
• Horizontal: entries are grouped by medication-name span, and each ground-truth entry is matched to the model entry with the best vertical F1 in its group. This also counts the relation between fields and the medication entry fig. 11 §6.2.

Statistical comparisons use an approximate randomization test with 1,000 resamples. A difference counts as significant if it exceeds the 95th percentile of the resampled differences app. D.1.

# Vertical metric, phrase level  [fig. 10] [app. C]
# Input:  documents, each with model entries and ground-truth entries
# Output: recall, precision, F1 over all documents

def vertical_metric(documents):
    total_tp = 0
    total_fp = 0
    total_fn = 0
    for doc in documents:
        # A field is (type, start_pos, end_pos); compare as sets  [fig. 10]
        model_fields = set(flatten_fields(doc.model_entries))
        gt_fields = set(flatten_fields(doc.gt_entries))
        total_tp += len(gt_fields & model_fields)
        total_fn += len(gt_fields - model_fields)
        total_fp += len(model_fields - gt_fields)
    # Ratios are taken once, over the summed counts  [fig. 10]
    recall = total_tp / (total_tp + total_fn)
    precision = total_tp / (total_tp + total_fp)
    f1 = 2 / (1 / precision + 1 / recall)
    return recall, precision, f1

Time cost is the total rater time per document, in minutes, including refinement tab. 2 §6.1. For the LLM base, rater time is zero by definition, and LLM inference time is reported separately app. B.

Hyperparameters

Data counts tab. 1:

Development-set results of the top configurations tab. 5:

Results

Author-reported. Setup: i2b2 2009 medication dataset, 216 test documents, PaLM 2 with no tuning §3.3 §5 §6. Quality is phrase-level tab. 2. Time is total rater minutes per document tab. 2.

Main comparison tab. 2:

Expert and non-expert base raters tab. 3. The test set is split by who made the base labels: expert base labels on 93 documents, non-expert base labels on 123 documents. The two "Base LLM" rows are evaluated on those respective subsets:

Significance of differences tab. 4 (differences are the first method minus the second; ** p<0.01, * p<0.05):

Per-field F1 tab. 7 (the paper does not state which split this uses):

Token-level metrics tab. 8:

Time model tab. 6 (linear mixed-effects regression of per-document time on counts of each correction type; ** p<0.01):

Headline findings §6:

• Refinement of LLM base labels reduces human time by 58% on average, from 26.67 to 11.32 minutes per document, with p < 1e-18 §6.1 tab. 2.
• LLM base labels and human base labels reach comparable quality after refinement §6.2 tab. 2 tab. 4.
• For expert base raters, LLM base plus refinement takes 11.97 minutes per document against 16.17 for expert base labels alone, a 26% reduction (p < 0.01) §6.3 tab. 3.

Paper vs code

No official code. The paper and arXiv listing link no repository. A GitHub repository search on the title and on "medication extraction LLM annotation" found nothing. No repository was cloned.

Pitfalls

• Which fields are labelled. The task description lists six fields, including reason §3.1. A footnote says the reason field was dropped, and the results tables cover five fields: name, dose, mode, frequency and duration tab. 1 tab. 7. The prompts still include reason and mode classes fig. 5 fig. 6 app. F. Decide explicitly whether reason is in scope.
• Chunk units and boundaries. Chunk sizes are given without a unit in the main text, and as characters only in the appendix §5.4 app. A.3. Splitting at line breaks and sentence ends is described but not specified exactly §5 step 1. Changing the unit changes which spans cross chunk borders.
• Ensemble merge rule. The ensemble text says the best configurations are combined §5.4. The pseudocode in the figure is a span-union merge: model 1 takes precedence on overlaps, and model 2 contributes only fields model 1 lacks fig. 9. A non-overlapping field from model 2 is dropped when model 1 has the same type. The alignment threshold for "partially overlapping" names is not stated.
• Metric details missing from the figures. The helper count_fields used in the horizontal metric is not defined fig. 11. The horizontal code matches each ground-truth entry to its best model entry, but the same model entry can match several ground-truth entries fig. 11. Model entries left unmatched within a name span are not counted as false positives in that code fig. 11. Check that your implementation matches the reported numbers before comparing with this paper.
• Vertical metric is exact. Fields match only on identical type and character positions fig. 10. Fuzzy matching in the resolver app. A.1 applies before this step, so the metric itself has no tolerance.
• Time comparison is not like-for-like. The 58% headline compares total rater minutes. The LLM base carries no rater time and its compute is excluded, and the authors state that LLM time cannot be directly compared to human time app. B §6.1. The pipeline also took iterative engineering effort, covering 10 IOB and 5 Direct Chunk prompt versions §5.4; that effort is not in the time figures.
• Refinement-only cost. Section 6.1 gives 11.3 and 9.1 minutes for refinement after LLM and after rater base labels. The 9.1 value is the refinement-only increment after a rater base (26.67 minus 17.60 in tab. 2); it appears in no table. The 11.3 value is the full human time, since the LLM base has no rater cost.
• Expert saving wording. Table 3's caption says expert base labels without refinement need 26% more time than LLM plus refinement. The table values (16.17 vs 11.97 minutes) give a 26% saving relative to the expert base, which is about 35% more time for the expert base. Use the raw minutes.
• Two "Base LLM" rows per subset. tab. 3 lists the LLM base twice, with different values, because each row is evaluated on a different document subset. Subset numbers are not directly comparable to tab. 2.
• Inconsistent reduction figures. §6.1 states 58%; the caption of tab. 2 states 57.6%. Both come from 26.67 and 11.32 minutes.
• Statistical test wording. The appendix says the p-values for the field-type comparison are below 0.5 app. D.2. This is very likely a typo for 0.05, but the text does not confirm it.
• Per-field split unknown. tab. 7 does not say which document set it uses.
• Prompts are not published. The appendix examples are described as synthetic and show one question-answer pair each app. F. The count of few-shot examples in production prompts, the rest of the prompt text, and the PaLM 2 variant are not stated. Implementers must write their own prompts and then evaluate them.
• Decoding details beyond temperature (top-k, top-p, stop sequences) are not stated.
• Numbering. Table and figure numbers here follow the order in the LaTeX source. The compiled PDF was not checked, so confirm the numbers there before citing them to readers.

Lineage

• Builds on: few-shot prompting §1 §5.2, citing Brown et al. (2020); IOB tagging, citing Ramshaw et al. (1999) §5.3; PaLM 2 as the base LLM §5.
• Compared against: the standard two-stage human pipeline, human base labels with expert refinement §4; prior active-learning work in clinical NLP is discussed but not compared quantitatively §2 §7.
• Superseded by: not stated.