Capabilities of Gemini Models in Medicine

In plain words

Med-Gemini is a set of AI models built on Google's Gemini family and tuned for medicine. The authors test them on exam-style medical questions, medical images, very long patient records and surgery videos. The work has three parts. The text model is taught to look things up on the web and to search again when its own answers disagree. The image models get extra training on medical pictures, and one version is given a special input for raw heart signals (ECG). For long records and videos, the models read the whole input at once, instead of in pieces.

The headline result is a new best score on the MedQA exam benchmark. The authors also had doctors re-check that benchmark's questions. About 7% of them turned out to be missing information, wrong, or open to more than one answer, so the authors report scores with those questions removed as well.

Most of the card is about results, which are the authors' own measurements on their own setup. The models and their code are not released. The one public artifact is a repository with the doctors' re-annotation data and an analysis notebook.

TL;DR

• Med-Gemini adapts Gemini 1.0 Ultra, 1.0 Pro, 1.5 Pro and 1.0 Nano to medicine with self-training, multimodal fine-tuning and long-context prompting §2.
• Its text model uses web search. An uncertainty-guided loop searches again only when sampled answers disagree §2.1.
• Author-reported: 91.1% on MedQA (USMLE), above the 90.2% of the prior best tab. 1.
• Author-reported: state of the art on 10 of 14 benchmarks (the paper's count); the multimodal comparison is not like-for-like (see Pitfalls).
• Models and model code are not released [back matter: Code Availability]. Only the MedQA re-annotation analysis is public.

Problem

LLMs show clinical knowledge but reason poorly under uncertainty, and they lack access to up-to-date medical information and tools §1. Fine-tuning on MedQA gives only the right letter, not expert reasoning, so models trained that way still make reasoning errors §2.1. Handling images, video and long health records is also limited, because of model context windows and the varied nature of medical signals §2.2 §2.3.

Method

The paper describes three capability tracks, each with its own model §2:

1. Reasoning with web search (Med-Gemini-L 1.0 and Med-Gemini-M 1.0). Med-Gemini-L 1.0 is Gemini 1.0 Ultra fine-tuned on reasoning data made by self-training. Med-Gemini-M 1.0 is Gemini 1.0 Pro fine-tuned for simpler language tasks: summaries and referral letters §2.1.
2. Multimodal fine-tuning (Med-Gemini-M 1.5 and Med-Gemini-S 1.0). Med-Gemini-M 1.5 is Gemini 1.5 Pro instruction-tuned on eight medical multimodal tasks from six datasets §2.2 tab. D1. Med-Gemini-S 1.0 adds a cross-attention encoder for raw 12-lead ECG waveforms to Gemini 1.0 Nano §2.2.
3. Long-context prompting (Med-Gemini-M 1.5 again). No new weights. The model is run in a long-context setting, with a two-step prompt for EHRs §2.3 and zero-shot prompts for videos §2.3.

Self-training with search (training data) §2.1 fig. 2

Expert clinical reasoning is costly to collect, so the authors generate it. Each training question gets two chains of thought: one without search results and one that includes results from a web search §2.1. The search queries are written by the model itself §2.1. Five hand-written expert demonstrations are used as seed examples for each chain type §2.1. Chains that lead to a wrong prediction are dropped before fine-tuning §2.1. The fine-tuned model then regenerates chains, and the loop repeats until performance stops improving §2.1.

The resulting MedQA-R set adds reasoning to MedQA. MedQA-RS adds search results as well §2.1 tab. C1. Both sets are 20354 examples in total tab. C1.

# Self-training with web search  [§2.1] [fig. 2]
# Input:  train questions with gold answers; base model; 5 seed demos per type
# Output: fine-tuned model, after the loop stops improving

def self_train(questions, answers, model, seed_demos, search):
    # Repeat generation and fine-tuning until performance saturates  [§2.1]
    while not saturated:
        training_chains = []
        for question, answer in zip(questions, answers):
            # Chain without search results  [§2.1]
            plain = model.chain(seed_demos.plain, question)
            # Model writes search queries, results go into the context  [§2.1]
            queries = model.search_queries(question)
            results = search(queries)
            searched = model.chain(seed_demos.searched, question, results)
            # Keep only chains whose final answer is correct  [§2.1]
            training_chains += keep_if_correct([plain, searched], answer)
        model = fine_tune(model, training_chains)
    return model

Uncertainty-guided search at inference (text tasks only) §2.1 §3.1

At inference the model answers in several rounds §2.1:

1. Sample several reasoning paths. The first round sees only the question. Later rounds also see the search results from the previous round §2.1.

2. Measure uncertainty as the Shannon entropy of the answer-choice distribution across those paths. A high value means the answers are spread out §2.1. The paper gives this in words, not as a formula; the formula below is the standard one. The log base is not stated.

   H = -\sum_{c} p_c \log p_c §2.1

3. If uncertainty is at or below a threshold, return the majority answer and stop. The threshold value is not stated §2.1.

4. Otherwise ask the model for three search queries that would settle the disagreement §2.1. Run them and add the results to the next round's prompt §2.1.

The search is applied to the three text benchmarks only. The authors leave the multimodal case to future work §2.1.

# Uncertainty-guided search at inference  [§2.1] [§3.1]
# Input:  question; model; n_paths; entropy threshold; n_rounds
# Output: answer letter

def answer_with_search(question, model, n_paths, threshold, n_rounds):
    # No retrieved evidence in the first round  [§2.1]
    evidence = []
    for iteration in range(n_rounds):
        # Sample several reasoning paths  [§2.1 step 1]
        responses = [
            model.generate(question, evidence) for _ in range(n_paths)
        ]
        choices = [parse_choice(response) for response in responses]
        # Entropy of the answer-choice distribution  [§2.1 step 2]
        uncertainty = shannon_entropy(choices)
        # Confident enough: return the majority answer  [§2.1 step 2]
        if uncertainty <= threshold:
            return majority_vote(choices)
        # Three queries that would resolve the disagreement  [§2.1 step 3]
        queries = model.make_queries(question, responses, count=3)
        # Retrieved text joins the prompt for the next round  [§2.1 step 4]
        evidence = evidence + web_search(queries)
    # Behaviour after the last round is not stated  [§3.1]
    return majority_vote(choices)

Multimodal fine-tuning §2.2 app. D.1

The fine-tuning mixture is a set of instruction tasks. Each dataset's weight in the mixture is roughly proportional to its number of training samples §2.2. Table D1 lists the eight tasks: Slake-VQA, Path-VQA and ROCO (VQA and captioning); PAD-UFES-20 (6-class dermatology); MIMIC-CXR (13-class and normal-vs-abnormal classification, and report generation); and ECG-QA tab. D1. The MIMIC-CXR results are deferred to a later paper app. D.1.

For PAD-UFES-20 the authors use three training variants: the original labels with clinical features, the same with augmentation (eight RandAugment operations), and an upsampled version for four minor classes app. D.1.

The ECG model (Med-Gemini-S 1.0) uses a cross-attention encoder in the style of Flamingo, reading a raw 12-channel ECG §2.2. Gemini is either frozen or unfrozen while the encoder trains app. D.3.

Long-context EHR and video §2.3 §3.3 app. E.1

For EHR search, the model reads all notes of one patient in one prompt. Step 1 pulls out every mention of the problem with a one-shot demonstration. Step 2 decides whether the problem is present, using only those mentions §2.3. The prompt is shown in fig. 8.

# Two-step long-context EHR check  [§2.3] [fig. 8]
# Input:  all notes of one patient; problem name (condition, symptom, procedure)
# Output: True if the patient has had the problem

def problem_present(notes, problem, model):
    # Step 1: retrieve every mention, one-shot prompt  [§2.3]
    mentions = model.find_mentions(notes, problem, one_shot=True)
    # Step 2: decide existence from the retrieved mentions  [§2.3]
    return model.decide_presence(problem, mentions)

For video, the model gets the whole clip with a task instruction and no examples §2.3. The two medical video localisation tasks use the MedVidQA dataset §3.3. The critical view of safety (CVS) task uses Cholec80-CVS §3.3.

Hyperparameters

Stated values are given with their source. Most training hyperparameters are absent from the paper.

Results

Author-reported throughout. Each result names the model variant, the dataset and the comparison. Parameter counts are not given.

Text-based tasks §4.1 tab. 1

Further text results:

• MedQA: accuracy rises from 87.2% to 91.1% over the search rounds §4.1 fig. 4. Self-training adds 3.2 points §4.1 fig. 4.
• NEJM CPC: search adds 4.0 points to top-10 accuracy §4.1. Per-specialty top-1 and top-10 numbers are in tab. C4. For top-10 in that table, search helps in internal medicine and pediatrics but not in neurology or psychiatry.
• GeneTuring: Med-Gemini-L 1.0 beats the prior best on 7 of 12 modules. Its averaged accuracy over the 12 modules is 53.3 against 48.6 tab. A1. Models that abstain are excluded from the accuracy numbers §3.1.
• MedQA relabelling (about 18 US primary care physicians, at least 3 ratings per question, bootstrapped) §4.1 app. C.2:
  • 3.8% of questions have missing information, 2.9% likely have a label error, and 0.7% are ambiguous (unanimous committees) §4.1. Together these are the 7.4% the Figure 1 caption reports fig. 1.
  • Each rater's agreement with the unanimous vote is 94.0% (missing information), 87.6% (label errors) and 94.6% (ambiguity) tab. C3.
  • Removing missing-information and label-error questions raises accuracy from 91.1% to 91.8% ± 0.2 §4.1.
  • With majority instead of unanimous votes, accuracy rises to 92.9% ± 0.38, discarding up to 20.9% of questions §4.1.
• Long-form expert side-by-side (clinician raters, paper's own tasks) §4.1.1 fig. 5 tab. C5:

Multimodal tasks §4.2 tab. 2

Further multimodal results:

• ECG-QA: frozen Gemini gives 57.7%; unfrozen gives 58.4%. The unfrozen setting beats an M3AE ECG model (57.6%) app. D.3.
• USMLE-MM by step: 89.5% (step 1, n=19), 92.9% (step 2, n=14), 100.0% (step 3, n=13) §4.2.
• The authors state a 44.5% average relative gain over GPT-4V across seven benchmarks §4.2 [abstract]. The card recomputes it below (Pitfalls).
• Dialogue examples (dermatology, radiology) are qualitative and single cases §4.2 fig. 6 fig. 7.

Long-context tasks §4.3 tab. 3

Further long-context results:

• EHR: on par with the heuristic baseline on F1. Rater agreement across the 200 examples is Jaccard 0.83 (all three agree), 0.915 (two or more agree) and Krippendorff's alpha 0.77 tab. E3.
• GPT-4 could not be run on the EHR set because the records exceed its public context limit §4.3. GPT-4V could not be run on most videos for the same reason §4.3.
• CVS: the appendix table gives Med-Gemini 50.0 against 0.290 for GPT-4V, a gap of 21 points tab. A1. The CVS paragraph says "21%" §4.3.
• Qualitative only: surgical action labelling on AVOS videos, a dialogue about a surgical video, EHR summarisation and dialogue, and a summary of 12 papers on the FTO locus §4.3 fig. 11 fig. 12 fig. 13 fig. 14.

Evaluation sets

Paper vs code

The only released code is a repository for the MedQA re-annotation analysis, at code_commit. The paper says model code and weights are withheld for safety reasons [back matter: Code Availability]. The repository is not linked from the paper. It was found through a GitHub search for "med-gemini" (first page of results, sorted by relevance). Its owner is the Google-Health organisation. Its README cites this paper and its authors, and it describes the MedQA re-annotation this paper reports. The paper links no project page. The card treats it as the authors' release on that basis. Drop the code fields if that is not accepted.

• Scope. The repository has no model, self-training, search, EHR or video code. It has the re-annotation data and one notebook medqa_relabelling.csv medqa_analysis.ipynb.
• Simulated predictions. The filtering analysis does not read model outputs. The notebook draws simulated predictions to hit a target accuracy of 0.911 medqa_analysis.ipynb. The cell says where to load real predictions. Its figure is therefore a rater-side simulation, not the paper's measured curve.
• Trial count. The notebook runs 10 bootstrap trials. The paper uses 1000 medqa_analysis.ipynb app. C.2.
• Consistent rules. Voting with unanimous (k=3) as the main setting and majority (k=2) for the appendix matches the paper medqa_analysis.ipynb app. C.2. The ambiguity rule (more than one option selected after the reveal) also matches medqa_analysis.ipynb medqa_analysis.ipynb.

No discrepancies were found for the model itself, because no model code is public.

Pitfalls

• Headline gain is stated two ways. The abstract and introduction give 4.6 points over Med-PaLM 2 §1. The results text gives 4.5 points §4.1. Quote the figure with its location.
• Filtered accuracies are not comparable with 91.1%. The 91.8% and 92.9% figures use subsets chosen by rater votes. The 92.9% setting drops up to 20.9% of questions §4.1.
• Relabelling numbers differ across sections. The results give 3.8%, 2.9% and 0.7%. The discussion rounds the first two to about 4% and 3% §5. Use the results values.
• The 44.5% is not a like-for-like average. The card recomputed it: the seven per-benchmark relative gains average 44.5%, using the GPT-4V numbers in tab. A1 for Slake-VQA, Path-VQA and PAD-UFES-20, and the GPT-4 number for ECG-QA. Those three Med-Gemini results are fine-tuned, and the GPT-4V baselines are zero-shot numbers the authors reproduced. The paper does not show this calculation.
• Percent vs points. The multimodal "8.7%, 13.1%, 2.6%" margins over GPT-4V are absolute point differences, not relative gains §4.2.
• The CVS number changes between tables. The main table gives 55.2 for Med-Gemini tab. 3. The appendix gives 50.0 tab. A1, and the text's "21%" matches 50.0 − 29.0. The appendix also writes the GPT-4V value as 0.290 (a fraction) next to percent values.
• CVS subset is unclear. GPT-4V was run on clips up to 5 minutes, and Med-Gemini was run on "the same filtered subset". ResNet3D used five folds. The paper does not say which clips the ResNet3D number used, or how many clips survived the filter app. E.1.
• EHR labels changed sampling balance. The paper samples 100 positive and 100 negative examples, then rater majority gives 121 positive and 79 negative app. E.1. The evaluation set is therefore not balanced.
• Long-context setting is undefined. The context length used is not stated §2.3. Table 3 marks its rows out-of-distribution, citing evaluation by in-context learning with no fine-tuning tab. 3. §2.3 says the same fine-tuned model is reused.
• Abstention changes the denominator. GeneTuring excludes abstentions from the accuracy calculation §3.1. Compare abstention rates in fig. 3 before comparing accuracies.
• Long-form review is uneven. Only the AVS task has each example rated by a single physician app. C.4. On the plain-language summary accuracy axis, experts are preferred 52% to 40% tab. C5. "As good or better more than half the time" holds for the overall axes, not for every axis.
• Search reproduction is hard. The search API, results per query, retrieval date, and how results are formatted are not stated §2.1. The self-training loop's stopping rule is also not stated §2.1.
• Self-training uses gold labels. The filter that drops wrong chains needs the correct answer, which the paper implies but does not state §2.1.
• Dataset and model artifacts are not public. The MedQA relabelling data is public [back matter: Data Availability] medqa_relabelling.csv. The model, MIMIC-CXR results app. D.1, and the fine-tuning data beyond the public benchmarks are not.
• Video comparisons are weak. GPT-4V is compared on frames sampled one per second, capped at 300 images app. E.1. Clips over five minutes were dropped for that comparison app. E.1, so it covers only short clips.
• Multimodal baselines differ across rows. ECG-QA is compared with GPT-4, not GPT-4V tab. 2. The PAD-UFES-20 row shows two splits, and the new patient-level split (78.8%) has no baseline tab. 2. The discussion notes a 7.1-point drop when retraining on that split §5.

Lineage

• Builds on: Gemini 1.0 and 1.5 (technical reports, cited in §1 and §2). The arXiv ids are not given in the source text, so not linked here.
• Prior work it improves on: Med-PaLM 2 (the prior text model on MedQA) and Med-PaLM M (the prior multimodal model) §2.2. AMIE (the prior NEJM CPC best) §4.1.
• Compared against: MedPrompt (GPT-4) tab. 1; GeneGPT, which the paper notes uses specialised web APIs rather than general search §4.1; GPT-4V tab. 2; RaNet and MutualSL on MedVidQA tab. 3; the annotation-and-aggregation heuristic on EHR tab. 3.
• Superseded by: not known from content already in this wiki.