Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

In plain words

Medical exam questions were, until recently, handled by models that were specially trained on medical text. This paper asks whether a general-purpose model can do as well if it is only given better prompts, with no extra training and no hand-written expert examples.

The answer it gives is a prompting recipe called Medprompt. For each question, it first looks up similar questions that it has already worked through and shows those as worked examples. Those worked examples were written by the same model, which explained its reasoning step by step, and only the ones that reached the right answer are kept. Then it asks the model the same question several times, each time with the answer options listed in a different order, and takes the answer that comes up most often. The authors report that this beats specialist models on the medical benchmarks they used, and they also test the same recipe on questions from other subjects.

TL;DR

Medprompt combines three general prompting pieces with GPT-4: nearest-neighbour few-shot examples, self-generated chain-of-thought rationales filtered by correctness, and a majority vote over option-shuffled calls. The paper reports state-of-the-art results on all nine MultiMedQA datasets, 90.2% on MedQA with the standard setting, and about 90.6% with 20 examples and 11 calls. These are author-reported numbers on multiple-choice benchmarks. The method uses no medical-specific components. Use it as a template for inference-time engineering, and budget for the preprocessing cost and the 5 calls per question.

Problem

Specialist medical results at the time came from domain-specific pretraining or fine-tuning, and from expert-written chain-of-thought prompts §1 §2. The paper argues that a general model such as GPT-4 can reach similar results through prompt design alone, and that a prior study used only simple prompting §1 §2. Test-time ensembles of specialist systems were expensive: Med-PaLM 2's ensemble refinement used as many as 44 inferences per question §2. The authors also state the risk of overfitting by prompt selection, since a prompt chosen by looking at benchmark results can look better than it is §3.2.

Method

Medprompt has two stages alg. 1 §4.4.

Preprocessing, once per pool item. Each question in the development pool is embedded with text-embedding-ada-002 §4.1 alg. 1. GPT-4 is asked for a chain of thought and a final answer §4.2. The question, its embedding, the chain of thought and the answer are kept only if the answer matches the gold label §4.2 alg. 1. Questions that fail this check are dropped from the pool entirely §4.4.

Inference, once per test question. The test question is embedded with the same model, and its k nearest pool items are retrieved by cosine similarity (distance = 1 minus cosine) alg. 1. The retrieved items are placed in the prompt as worked examples, each with its stored chain of thought §4.4. The test question and its answer options come last §4.4. The procedure repeats for n calls, shuffling the answer options before each call §4.3 alg. 1. The final answer is the most frequent answer across the calls alg. 1.

The three components are:

1. Dynamic few-shot selection §4.1. Examples are chosen per test question instead of fixed once, which the paper argues needs no fine-tuning §4.1.
2. Self-generated chain of thought §4.2 fig. 2 fig. 3. GPT-4 writes the rationales for the pool, using a template with a question, options, a chain-of-thought section and a line "Therefore, the answer is" fig. 3. The paper reports that these rationales are longer and more step-by-step than the expert-written ones used by Med-PaLM 2 §4.2 §5.2.
3. Choice-shuffle ensemble §4.3. The option order is shuffled before each reasoning path to reduce position bias, and the most consistent answer is selected §4.3. The same shuffling is used when generating chains of thought for pool items, where only correct chains are kept §4.3.

Standard Medprompt uses 5 retrieved exemplars and 5 ensemble calls §4.4 alg. 1. Increasing these to 20 exemplars and 11 calls gives a further 0.4 points on MedQA §4.4 §5.2 tab. 1.

# Medprompt  [alg. 1]
# Input:  development pool, test question Q with answer options
# Output: final answer

def build_pool(pool, embed, generate_cot):
    # Embed each pool question once  [§4.1] [alg. 1]
    entries = []
    for question in pool:
        vector = embed(question.text)
        # Keep the pool item only if the generated answer is correct  [§4.2]
        chain, answer = generate_cot(question)
        if answer == question.gold:
            entries.append((question, vector, chain, answer))
    return entries

def medprompt_answer(query, entries, embed, llm, n_shot, n_calls):
    # Embed the test question with the same embedding model  [alg. 1]
    query_vector = embed(query.text)
    # Retrieve the n_shot most similar items by cosine similarity  [alg. 1]
    shots = nearest_by_cosine(query_vector, entries, n_shot)
    votes = []
    for call in range(n_calls):
        # Shuffle the answer options before each call  [§4.3] [alg. 1]
        order = shuffled_option_order(query.options)
        # Ask for a chain of thought and an answer, given the shots  [§4.4]
        choice = llm(shots, query.with_options(order))
        # Map the chosen position back to the original option  [alg. 1]
        votes.append(order[choice])
    # Majority vote over the calls  [alg. 1]
    return most_common(votes)

Evaluation protocol. For each benchmark, 20% of the data is held out as an "eyes-off" split that is not looked at during prompt development §3.2. The paper reports results on the held-out split separately §5.1. The benchmark sizes it uses are: MedQA (US, 4 options) 1273 questions, MedMCQA dev 4183, and PubMedQA reasoning-required 500 §3.1. The nine MultiMedQA datasets are MedQA, PubMedQA, MedMCQA and six medical subsets of MMLU §3.1.

Hyperparameters

Results

Author-reported. Setup: GPT-4 through API; benchmarks are multiple-choice MultiMedQA datasets §3.1. Flan-PaLM 540B and Med-PaLM 2 numbers are taken from prior work and use each model's best-of-several strategy per dataset; GPT-4 numbers use one strategy for all datasets tab. 1.

• MedQA reaches 90.6% with 20 exemplars and 11 ensemble calls; the table's 90.2% is the standard 5 and 5 setting tab. 1 §5.2.
• Headline claims: state-of-the-art on all nine datasets §5 tab. 1; a 27% reduction in MedQA error over the best earlier specialist result [abstract]; an order of magnitude fewer calls than Med-PaLM 2 [abstract] §2.

Overfitting check §5.1 fig. 5. Average accuracy on the tuning split is 90.6% and on the held-out split 91.3%. The held-out score is higher on 6 of 9 datasets. The paper reads this as no sign of overfitting from prompt engineering.

Ablation on MedQA §5.2 fig. 6 tab. 2. Starting from a 5-shot baseline, each component was added in turn. Chain of thought gave +3.4 points, and few-shot selection and choice shuffling each gave +2.2 points. The order of addition is subjective, and the paper does not compute Shapley values §5.2. Replacing the expert chain of thought by GPT-4's own, with 5 fixed shots and no ensemble, gives 83.8 vs 86.9 on MedQA tab. 2.

Other domains §5.3 fig. 7. Medprompt with no changes was tested on six MMLU subjects (electrical engineering, machine learning, philosophy, professional accounting, professional law, professional psychology) and two NCLEX-style nursing datasets §5.3. The average gain over zero-shot prompting is +7.3 points, against +7.1 points on the MultiMedQA datasets §5.3.

Paper vs code

The official code is in the microsoft/promptbase repository, MIT licence, at commit bf5d0dcc7f92650e50f351bf3878efbeb6dae385. The paper does not link it. The repository's README cites this paper and describes the Medprompt method README.md. It is maintained by Microsoft, where the authors work. No other repository was used.

• Option order. The paper says options are shuffled §4.3 alg. 1. The code uses a deterministic "plain hunt" sequence of adjacent swaps. The first order is the identity, and each later order is one swap pattern applied to the previous one fewshot_cot_as_conversation_ensemble.py fewshot_cot_as_conversation_ensemble.py.
• Calls in parallel. The paper says the 5 calls are parallel §4.4. The code calls them in a sequential for loop fewshot_cot_as_conversation_ensemble.py.
• Exemplar rationales. The paper says GPT-4 wrote the rationales for the pool, with option shuffling §4.2 §4.3. The pipeline calls zero_shot_cot.py, which states at its top that it "is not what generated the reported results" zero_shot_cot.py. The pipeline runs one zero-shot pass per pool item without shuffling submit_mmlu_fewshot_knn_cot.py.
• Prompt format. The paper shows a "## Question / ## Answer / Therefore, the answer is" template fig. 3. The code uses a system instruction, then chat turns labelled "Explanation" and "Final Answer", with numbered options fewshot_cot_as_conversation_ensemble.py.
• Example pool. The text says training set §4.1. The code takes exemplars from the MMLU validation split knn_fewshot_cot_ensemble_config.yaml.
• Vote and selection. The paper uses the mode alg. 1. The code maps each chosen position back to the original option and keeps the lowest index among tied maxima fewshot_cot_as_conversation_ensemble.py.
• Datasets. The pipeline fetches MMLU only knn_fewshot_cot_ensemble_config.yaml. The repository has no MedQA, MedMCQA or PubMedQA loader. The string "medqa" only appears in src/promptbase/mmlu/prompt_templates.py. So the MultiMedQA numbers cannot be reproduced from this code alone.
• Best setting. The 20-exemplar, 11-call setting is not in the repository. Its defaults are 5 and 5 knn_fewshot_cot_ensemble_config.yaml fewshot_cot_as_conversation_ensemble.py.
• Beyond the paper. The README also describes "Medprompt+", a portfolio with extra direct-answer prompts and logprob-based screening [README.md]. This is not in the paper and is not covered here.

Pitfalls

• Pool filtering. Wrong rationales are dropped, so the pool only contains questions the model already answers correctly §4.2 alg. 1. The paper says a rationale can still reach the right answer by faulty reasoning, and the filter does not catch that §4.2. The pool size after filtering is not stated.
• Cost. Preprocessing needs one embedding and one chain-of-thought call per pool item alg. 1. Inference uses 5 calls per question in the standard setting, each with 5 long exemplars alg. 1 §4.4. Check the total token budget before using 11 calls or 20 exemplars.
• Temperature not stated. If sampling is at temperature 0, the 5 calls differ only by option order, so the ensemble's diversity comes only from the order. This is inference from the method, not a claim by the paper. Set the temperature explicitly. The paper does not say what it used.
• Option-order mapping. The vote must map each answer back to the original option index. The paper states this implicitly alg. 1. The code does this with current_permutation[selected] fewshot_cot_as_conversation_ensemble.py.
• Two-option questions. For two options, the code's permutation generator yields only four orders, and only two of them differ. A fifth next() call would then raise StopIteration fewshot_cot_as_conversation_ensemble.py fewshot_cot_as_conversation_ensemble.py. This is from a hand trace, not a run. For four options, which the benchmarks here use, the generator gives 8 orders, enough for 5 calls.
• Tie-breaking. The mode is not defined for ties alg. 1. The code's rule is the lowest original index. A different tie rule can change results on uncertain questions.
• Eyes-off gap. The held-out average is 0.7 points higher than the tuning average §5.1. The split reduces but does not remove tuning bias: the same design choices were made on the other 80%.
• Benchmark contamination. The paper says contamination during training cannot be excluded §6.
• Medical claims. The paper says benchmark performance does not show real-world clinical value §6.
• Ablation numbers. Only +3.4, +2.2 and +2.2 are given in text. The other bar values appear only in fig. 6.
• Author-reported results. The paper does not state the GPT-4 snapshot or date used, and the numbers depend on the API version.

Lineage

• Builds on: chain-of-thought prompting (arXiv:2201.11903) §2.2 and self-consistency voting over sampled chains (arXiv:2203.11171) §2.2. Medprompt replaces sampling with option-shuffled calls and adds retrieved examples §4.3.
• Builds on the authors' earlier study of GPT-4 on the same medical benchmarks with simple prompting (Nori et al., 2023; no arXiv id given in the paper) §1 §2.
• Compared against: Med-PaLM (arXiv:2212.13138) §2 tab. 1 and Med-PaLM 2 (arXiv:2305.09617) §2 tab. 1.
• Superseded by: not stated in the paper.