wikillm

Paper · 2021-09-09 · cs.CL, cs.AI, cs.LG

Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization

Labels medical dialogue snippets with GPT-3 instead of doctors: K=10 summaries from random 21-example prompts, keeping the one covering the most medical concepts. Training PEGASUS on 6400 such labels, built from 210 human examples, beats 6400 human-labelled summaries on concept and negation F1 but not ROUGE-L. DR-SUM is worse on all three. Author-reported.

Text summarizationIn-context learningFew-shot learningSynthetic training datamedical-summarizationsynthetic-datafew-shot-promptingoutput-ensembling

arXivPDFMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

After each telehealth chat, a doctor has to write a summary. A software summarizer can learn to do this from doctor-written examples, but those examples are slow and costly to make, and patient chats cannot simply be sent to an outside service. This paper asks whether a large general-purpose text model can write the training examples instead.

Such a model is not reliable on its own. It often leaves out medical details or gets yes and no statements wrong. But its output changes with the examples it is shown. The method asks it for several summaries, each primed with a different set of example pairs, and keeps the one that mentions the most medical terms from the chat. The kept summaries become training data for a smaller summarizer that the team runs itself. The authors argue that the chats sent to the outside service can be chosen with privacy rules in mind, and that the trained summarizer then needs no outside service at use time.

Mixing these machine-written examples with the doctor-written ones helped in most settings the paper tested. The results are the authors' own. The two summarizers they train gain on some medical metrics but not on all of them.

TL;DR

  • The paper builds GPT-3-ENS: it primes GPT-3 with K=10 random sets of 21 labelled snippet-summary pairs, then keeps the candidate summary whose medical concepts best cover the snippet's §4.2 alg. 1 §7.
  • GPT-3-ENS labels unlabelled snippets, and the labels train PEGASUS or DR-SUM. A 6400-snippet set built from 210 human examples is compared with 6400 human-labelled snippets §5 §7.1.
  • Author-reported: PEGASUS trained on the GPT-3 labels beats the human-data baseline on Concept F1 and Negation F1 but not ROUGE-L at that size. DR-SUM is below its baseline on all three metrics at that size tab. 2.
  • Mixing human and GPT-3 labels improves most metrics over human data alone, but not for every mix ratio tab. 4 §7.2.
  • The selector depends on an in-house medical concept extractor that is not released §4.2.

Problem

Medical dialogue summaries must be coherent and must keep the medically relevant facts §1 §2. Training a summarizer needs many expert-written summaries, which are costly and slow to produce, and labelled patient data may not be shared because of privacy rules such as HIPAA §1 §2. Fine-tuning pretrained summarizers still needs hundreds to thousands of labelled examples §1. Using an external low-shot model directly at inference is blocked when patient data cannot leave the organisation, and a model owned by a third party cannot easily take the practitioner corrections that improve it over time §2.

The paper also reports that GPT-3 used directly, primed with a few manually labelled pairs, misses medical concepts, sometimes gets affirmations wrong, and repeats the doctor's questions §4.1 tab. 1.

Method

Overview. The approach has three parts: (1) a prompt format and an ensemble selector that make GPT-3 usable as a labeller, GPT-3-ENS §4.2; (2) a data generator that runs GPT-3-ENS over unlabelled snippets to build a training set §4.3 §5; (3) training two off-the-shelf summarizers on that set, alone or mixed with human-labelled data §7.1 §7.2. fig. 1 summarises the idea in a diagram that is described here, not reproduced.

  1. Snippets. Each dialogue is cut into snippets at the turns between two consecutive physician questions. A snippet runs from two to ten turns. Snippets with no history-taking content (greetings, purely educational talk) are excluded from annotation §5.
  2. Human labels. Medical doctors from the same telehealth platform write a summary per snippet, as for a clinical note. This gives 6900 snippet-summary pairs, of which 500 form the test set T and the other 6400 form the pool H6900 − T §5.
  3. Priming format. Each priming example is written as the snippet, then a summarised marker, then the summary, then a stop marker. Turns in the snippet are separated by a separator token. The appendix shows this for two examples app. A. The paper uses N = 21 examples per prompt, the largest number that fits the 2048-token context window of the API §4.1 §7.
  4. GPT-3 summarizer. GPT-3 is primed with N examples and asked to summarise a new snippet, with no gradient updates §1 §4.1 (Brown et al., 2020).
  5. Medically aware ensemble, GPT-3-ENS. The selector below runs K trials, each with a fresh set of N examples sampled from the pool L, and returns the summary whose medical concepts overlap the snippet's most. The concept extractor is the authors' in-house tool, built on UMLS, which covers symptoms, disorders, lab tests and medications §4.2. The paper says any recognizer with that coverage can be used §4.2.
Pseudocode
# GPT-3-ENS: medically aware ensemble summarizer  [alg. 1] [§4.2]
# Input:  snippet (dialogue text); K ensembling trials; N examples per
#         prompt; pool L of labelled examples; recognizer (concept
#         extractor); gpt3 (primed model call)
# Output: one summary of the snippet
def gpt3_ens(snippet, K, N, pool, recognizer, gpt3):
    # Medical concepts in the snippet: the reference set C*  [alg. 1]
    reference = recognizer(snippet)
    best_summary = None
    best_recall = -1.0
    for trial in range(K):
        # Draw N labelled examples from the pool for this prompt  [§4.2]
        examples = sample_without_replacement(pool, N)
        # Summary of the snippet from GPT-3 primed with those examples  [alg. 1]
        summary = gpt3(examples, snippet)
        # Concepts found in this candidate summary  [alg. 1]
        found = recognizer(summary)
        # Share of the reference concepts the candidate covers  [alg. 1] [§4.2]
        recall = len(found & reference) / len(reference)
        # Keep the first candidate with the highest recall; ties not stated  [alg. 1]
        if recall > best_recall:
            best_recall = recall
            best_summary = summary
    return best_summary
  1. Synthetic dataset. The paper writes a dataset of size p built with K trials as GCF_p^K. Building it needs K × N labelled examples for priming, sampled from H6900 − T without replacement, with the same random seed across comparisons, so with K = 10 and N = 21 all 210 examples are used §5 §7.1. The pool of priming examples is fixed at 210 in every experiment that calls GPT-3 §7.
  2. Training. Two summarizers are trained on the synthetic set, alone or mixed with human labels: PEGASUS (abstractive) and DR-SUM (hybrid, the 2M-PGEN variant of Joshi et al., 2020) §7 §1. A mixing ratio α gives α times the number of human examples in synthetic labels, with α in {0.5, 1, 2, 3} §7.2.
  3. Evaluation. Human test set T (500 snippets). Metrics: ROUGE-L F1 with the default package configuration, Concept precision, recall and F1, and Negation F1 from NegEx §6.1. Doctors also grade summaries and pick the best one per snippet §6.2 fig. 2 fig. 3 fig. 4.

Hyperparameters

NameValueWhereNotes
Priming examples per prompt, N21§4.1 §7"Typically" in §4.1; fixed to 21 in §7 because of the 2048-token window
Ensembling trials, K10§7 §7.1Stated as the default for all experiments unless otherwise specified; no K sweep appears in the compiled text
Labelled pool for priming, L210 random examples from H6900 − T§7210 = 21 × 10
Human-labelled snippets6900 pairs§5Source of H6400 after holding out T
Test set, T500 snippets§5Used in all evaluations
Sampling of priming examplesWithout replacement; same seed across comparisons§4.2 §5Trials draw disjoint examples in the dataset construction
GPT-3 model variantnot stated§7Accessed through the beta OpenAI API
Context window2048 tokens§7
Max response length128 tokens§7
Temperature0.6§7
Presence penalty / frequency penalty0 / 0§7
Ensemble selection scoreRecall of reference concepts, |C_i ∩ C*| / |C*|alg. 1 §4.2Ties and empty C* not stated
Mixing ratio, α0.5, 1, 2, 3§7.2α = synthetic / human examples
Synthetic dataset sizes6400, 12800, 25600§7.1Same 210 priming examples in all sizes §7
DR-SUM pretrainingJoshi et al. (2020) settings on CNN/DailyMail§7Values not restated in this paper
DR-SUM fine-tuning batch size16§7
DR-SUM source / response max tokens400 / 200§7
DR-SUM gradient clippingmax norm 2.0§7
DR-SUM epochs2§7
DR-SUM learning rate / optimiser0.15 / Adagrad§7
PEGASUS checkpointpegasus-cnn_dailymail (Hugging Face)§7
PEGASUS effective batch size256§7How the effective size is reached is not stated
PEGASUS source / response max tokens512 / 128§7
PEGASUS epochs2§7
PEGASUS learning rate / optimisernot stated / Adafactor at Hugging Face defaults§7
Decoding (both models)Beam size 4§7
Warm-up, LR schedule, seeds for trainingnot stated

Results

Author-reported. Setup: snippets from the authors' telehealth platform (internal data; no public release is mentioned in §5); test set of 500 snippets §5; baselines are PEGASUS and DR-SUM trained on 6400 human-labelled snippets §7.1. Hardware and number of runs are not stated, and no variance is reported in the compiled text.

Synthetic labels only tab. 2:

ModelTrain dataNegation F1Concept F1ROUGE-L F1
PEGASUSH6400 (human)21.0935.9655.59
PEGASUSGCF 6400, K=1028.8940.0253.43
PEGASUSGCF 12800, K=1026.7040.2156.66
PEGASUSGCF 25600, K=1028.6140.5858.44
DR-SUMH6400 (human)26.7539.9552.70
DR-SUMGCF 6400, K=1024.2937.5548.47
DR-SUMGCF 12800, K=1026.6638.4949.18
DR-SUMGCF 25600, K=1026.0839.4750.85

Headline: 210 human examples, used to prime GPT-3-ENS, give 6400 synthetic labels. The paper calls the result comparable to or better than 6400 human labels. On this table, that holds for PEGASUS on the two medical metrics; PEGASUS at 6400 is lower on ROUGE-L, and DR-SUM at 6400 is lower on all three tab. 2 §7.1.

Mixed human and synthetic data tab. 4:

ModelTrain dataNegation F1Concept F1ROUGE-L F1
PEGASUSH640021.0935.9655.59
PEGASUSα = 0.5: H6400 + GCF 320030.1443.4962.45
PEGASUSα = 1: H6400 + GCF 640030.7043.7360.63
PEGASUSα = 2: H6400 + GCF 1280029.4341.0259.85
PEGASUSα = 3: H6400 + GCF 2560031.9344.6861.05
DR-SUMH640026.7539.9552.70
DR-SUMα = 0.5: H6400 + GCF 320027.5140.4653.39
DR-SUMα = 1: H6400 + GCF 640027.1840.3651.00
DR-SUMα = 2: H6400 + GCF 1280027.1940.6853.07
DR-SUMα = 3: H6400 + GCF 2560026.3339.8952.29

For PEGASUS every mix beats the human baseline on all three metrics. For DR-SUM, α = 0.5 and α = 2 beat it on all three, α = 1 is lower on ROUGE-L, and α = 3 is lower on all three tab. 4 §7.2. The paper notes that the claim is that mixing beats human data alone, not that a larger α is better §7.2.

Doctor evaluation (figures only; no counts appear in the text):

  • On snippet summaries from GPT-3 with and without ensembling, doctors picked the GPT-3-ENS summary as best more often fig. 2. The text calls the difference "significant" but names no test §4.2.
  • For PEGASUS, doctors rate the model trained on GCF 6400 above the one trained on H6400, and prefer the mixed model (α = 0.5) over both fig. 3 §7.2.
  • For DR-SUM, the mixed model is again preferred over human-only and synthetic-only models fig. 4 §7.2.
  • Doctors graded coverage as All (100%), Most (at least 75%), Some (at least one fact, under 75%) or None (0%) §6.2.

Qualitative examples: tab. 1 (GPT-3 alone versus GPT-3-ENS on four snippets) and tab. 3 (outputs of the three trained models on two snippets).

Paper vs code

No official code.

What I searched: the arXiv abstract page lists no code, project page or repository. The paper links only a ROUGE package, the Hugging Face PEGASUS checkpoint and the UMLS and OpenAI sites §6.1 §7. GitHub API searches for "medically aware gpt-3" returned five unrelated repositories; "gpt-3-ens summarization", "gpt3 medical summarization" and "dialogue summarization gpt-3 synthetic data medical" returned nothing. The Curai organisation has three repositories (curai-research, medical-question-pair-dataset, hapi-fhir-starter). The archived curai-research README covers other Curai papers and does not mention GPT-3 or ensembling; it lists a different ML4H 2021 PEGASUS summarization study that I could not identify with this paper, so I did not use it. The profile of a first author matching the name "Bharath (Jai) Chintagunta" has one unrelated repository. None of these is an official implementation of GPT-3-ENS.

Pitfalls

  • The concept extractor decides the result, and it is not released. GPT-3-ENS selects by overlap with the in-house UMLS-based recognizer §4.2. The paper says any recognizer with the same coverage can be used, but gives no alternative and no test of sensitivity §4.2. Concept F1 and the selection step will not match the paper unless you build a similar extractor. Expect different numbers.
  • Sampling across trials. Algorithm 1 draws N examples per trial from L alg. 1. The text says sampling is without replacement §4.2 and that the k trials for a dataset use no reused examples §5. With L = 210, N = 21 and K = 10, that means every trial uses a disjoint block. Implement it this way; the algorithm box alone does not say so.
  • Edge cases in the selector. The recall score divides by |C*|, the concept count of the snippet. The paper does not say what happens when the snippet has no concepts, and it does not say how ties are broken alg. 1. Decide both before running at scale.
  • Priming overlap with training data. The priming examples come from the same pool as the labelled snippets used for training §5 §7. The paper does not say whether the snippets that GPT-3 labels exclude the 210 priming examples (an inference from the stated sizes: the 6400-snippet set can cover the whole pool). Check this before comparing numbers.
  • Privacy claim depends on process. The paper says the data sent to GPT-3 is small and fixed, and that experts choose dialogues "following privacy protocols" §2 §4.3. But the snippets to be labelled must also be sent to the API §4.3. The paper gives no procedure for selecting or de-identifying them. Do not read the privacy argument as covering the labelling inputs without a separate check.
  • Model version and cost. The GPT-3 model variant is not stated, and the paper used a beta API §7. The exact labeller cannot be recovered, and any rerun uses a different model. Each snippet costs K = 10 model calls and K concept extractions (inference from alg. 1). Cost in money or time is not stated.
  • Prompt at inference time. The appendix shows the priming examples and their markers, but not the exact query suffix or stop handling for a new snippet app. A. Infer both (a summary marker after the snippet, a stop at the marker that ends a summary) and test them.
  • Doctor evaluation. Doctor results appear only in figures fig. 2 fig. 3 fig. 4. The text names no statistical test and no number of raters or snippets §6.2 §4.2.
  • Mixed results. The headline "comparable (or better)" holds for PEGASUS on Negation and Concept F1 only. ROUGE-L is lower at 6400 for PEGASUS and lower on all three metrics for DR-SUM tab. 2. Report the table, not the headline.
  • Single runs. The compiled text reports single numbers with no variance and no repeated seeds §7 tab. 2 tab. 4. Differences of about one point (for example DR-SUM ROUGE-L 52.70 against 53.07) are within what a rerun could move.
  • Concept F1 definition. Concept precision and recall are written as sums over snippets §6.1. Concept F1 is named but not written out (the harmonic mean is implied; inference). An example with no detected concepts in the snippet and summary gets F1 = 0 §6.1, footnote.
  • Snippet filter. Snippets with no history-taking content are dropped during annotation §5. The paper does not say whether the same filter applies to the GPT-3 labelling inputs.

Lineage

  • Builds on: GPT-3 few-shot priming (Brown et al., 2020; arXiv:2005.14165) §1 §4.1; Dr. Summarize, which supplies the snippet approach and the 2M-PGEN variant (Joshi et al., 2020; arXiv:2009.08666) §1 §5 §7; PEGASUS (Zhang et al., 2019; arXiv:1912.08777) §3 §7. Dr. Summarize shares three authors with this paper (Katariya, Amatriain, Kannan) [bibliography].
  • Compared against: PEGASUS and DR-SUM trained on 6400 human-labelled snippets §7.1; GPT-3 primed without the ensemble §4.1 tab. 1 fig. 2.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization” by Bharath Chintagunta, Namit Katariya, Xavier Amatriain, Anitha Kannan (arXiv:2110.07356), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.