Towards Generalist Biomedical AI

In plain words

Doctors combine many kinds of information: notes, lab results, scans and, sometimes, genetic data. Most medical AI systems handle one kind of input for one narrow job, such as reading mammograms. This paper asks whether a single model can read several kinds of medical data and answer several different kinds of question with the same set of weights.

The authors first build a test set, MultiMedBench, that mixes medical questions, scans, X-ray reports and genetic variant calls. They then take an existing general-purpose model that already accepts text and images and fine-tune it on that mix. Every task is phrased as a text instruction, and the model writes its answer as text. That turns classification, question answering and report writing into one kind of job.

The model is competitive with or better than specialist systems on many of the tasks. It also shows some abilities that were not trained directly, such as handling a tuberculosis question it never saw as a label. Doctors reviewed its chest X-ray reports and found them encouraging, but the authors say much more validation is needed before real-world use.

TL;DR

• Med-PaLM M is one multimodal model, fine-tuned from PaLM-E at 12B, 84B and 562B parameters, that handles 14 biomedical tasks with one set of weights §1 §4.2 tab. A.2.
• MultiMedBench, the benchmark introduced for it, spans text QA, radiology and pathology VQA, chest X-ray and dermatology and mammography classification, radiology summarization, chest X-ray report generation and genomic variant calling §3 tab. 1.
• Author-reported: near or above prior SOTA on most tasks; the 84B model is best on most chest X-ray report metrics tab. 2 tab. 3.
• No code or weights are released; the LLMs are not open-sourced [Code Availability statement after §9].

Problem

Single-task, single-modality specialists cannot use other data sources, cannot explain a prediction and cannot take part in a dialogue. They also output only a fixed set of labels §1. The authors add that the field lacked a multimodal medical benchmark to train and compare generalist models §1 §2.3. Prior multimodal biomedical models were either pretrained only, needing task-specific fine-tuning §2.2, or limited to a few VQA datasets §2.2.

Method

Overview. Med-PaLM M starts from PaLM-E, a multimodal model made of a PaLM language model and a ViT image encoder that feeds image tokens into the language model §4.1. The authors fine-tune all parameters on MultiMedBench with instruction-style prompts and text targets, and they do not add task-specific heads §4.2. The paper has no numbered equations and no algorithm box; the pseudocode below is reconstructed from the text.

Steps

1. Benchmark mix. MultiMedBench has 14 tasks over 12 datasets and over one million samples §3 tab. 1. Per-dataset details are in the benchmark table below and in app. A.1.1 and app. A.1.2.
2. Images. Each image is resized to 224 x 224 x 3, keeping the aspect ratio and padding if needed. Grayscale images are copied into three channels §4.2. Per-task augmentation and class balancing are in app. A.1.2.
3. Prompt. Each prompt holds a task instruction, optional context (for example the reason for a chest X-ray study and its view, or a dermatology patient history), and the question. Classification tasks list every label as an answer option, and the model generates the answer text §4.2 fig. 2. Generative tasks train on the reference response §4.2.
4. One-shot exemplar. For most tasks, a text-only exemplar pair is placed before the question. The exemplar keeps its answer, but its image is replaced by a dummy text placeholder (the paper writes it as "img" in angle brackets). The authors say this saves compute on single-image training and avoids cross-attention between image tokens from several images §4.2 fig. 2. Which tasks use 0, 1 or 2 shots is in tab. A.1.
5. Vision tokens. The ViT maps each image to 256 tokens app. A.2.2. The vision encoder is a 4B ViT for the 12B model and a 22B ViT for the 84B and 562B models §4.1. Image tokens are interleaved with text tokens. Each fine-tuning input has at most one image §4.2.
6. Mixture sampling. Ratios are set empirically to be roughly proportional to each dataset's sample count, with at least one sample per task in every batch §4.2 tab. A.1.
7. Optimizer. Adafactor with momentum beta1 = 0.9, dropout 0.1, constant learning rate. Learning rate and batch size depend on model size §4.2 tab. A.2.
8. Inference. The decoder generates text. Classification is also open-ended generation, so the label is the generated string §3.

# Build one training example  [§4.2] [fig. 2]
# Input:  instruction, exemplar (or None), context, question, target, image
# Output: interleaved input sequence and target text

def build_example(instruction, exemplar, context, question, target, image):
    # Resize to 224x224x3 keeping aspect ratio, pad if needed  [§4.2]
    pixels = resize_and_pad(image, size=224) if image is not None else None
    # Start with the task instruction  [§4.2]
    parts = [instruction]
    if exemplar is not None:
        # Text-only exemplar; its image becomes placeholder text  [§4.2]
        parts = parts + [exemplar.text_with_img_placeholder, exemplar.answer]
    # Context (e.g. view, reason for study) followed by the question  [§4.2]
    parts = parts + [context, question]
    return interleave(parts, pixels), target

# One fine-tuning step; all weights are updated  [§4.2] [app. A.2.2]
# Input:  batch sampled with the mixture ratios  [tab. A.1]
# Output: updated weights

def train_step(weights, batch):
    # Each image becomes 256 ViT tokens  [app. A.2.2]
    image_tokens = [vit(weights, image) for image in batch.images]
    # Image tokens are interleaved with the text embeddings  [§4.2]
    sequence = interleave_embeddings(batch.text, image_tokens)
    # Decoder predicts the target response  [§4.1] [§4.2]
    logits = decoder(weights, sequence)
    # Loss form on target tokens is not stated  [§4.2]
    loss = target_token_loss(logits, batch.targets)
    # Adafactor, beta1 = 0.9, dropout 0.1, constant learning rate  [§4.2]
    return adafactor_update(weights, loss, beta1=0.9)

Benchmark: MultiMedBench

Tasks span five task types and seven modalities §3 tab. 1. The split sizes below come from the appendix; "official" means the dataset's own split.

Data mixture tab. A.1. Language-only tasks are under 15% of the mix.

Evaluation

• Generalist benchmark §5.1: test split of each task, compared with prior SOTA specialists and with the PaLM-E 84B baseline without medical fine-tuning. Same few-shot setup as training.
• Zero-shot generalization §5.2: tuberculosis (TB) detection on the Montgomery County chest X-ray set (138 frontal films, 80 normal, 58 TB). The model is prompted with a text-only exemplar, and the image is omitted. The exemplar is hand-written, not drawn from training data. Two-view report generation tests a view setup that was not in training. A positive-transfer ablation drops the MIMIC-CXR classification task from the 84B mixture.
• Radiologist evaluation §5.3: 246 chest X-ray cases from the MIMIC-CXR test split, one image per study, with studies that mention several views or prior exams excluded. Four thoracic radiologists based in India took part. Side-by-side ranking of four candidate findings (reference plus the three model sizes) was done on all 246 cases, each by one randomly chosen rater. The independent evaluation had all four raters annotate findings from each model size on every case. A pilot of 25 cases calibrated the raters §5.3.

Hyperparameters

Source: tab. A.2 and text in §4.1, §4.2 and app. A.2.2 unless marked.

Results

Author-reported throughout. Setup: MultiMedBench test splits; comparison against prior SOTA specialist models and the PaLM-E 84B baseline without medical fine-tuning §5.1. "Best" means the best of the three Med-PaLM M sizes for each cell tab. 2. Bold in the source marks the best value in each row; it is not reproduced here. SOTA citations are shown as in the paper.

Table 2: benchmark results tab. 2. SOTA cites: Med-PaLM 2 [singhal2023towards] for MedQA/MedMCQA/PubMedQA; van2023radadapt for MIMIC-III; bazi2023vision and van2023open for VQA; nicolson2022improving, miura2020improving and jeong2023multimodal for CXR; rammuni2022effective and wantlin2023benchmd for classification; panambur2022effect for CBIS calcification; poplin2018deepvariant for variant calling.

Table 3: scaling across sizes tab. 3. Each size was finetuned and evaluated on the same tasks.

Zero-shot TB detection tab. 4. Accuracy on Montgomery County. The SOTA is a model ensemble trained on all 138 samples. Med-PaLM M used 0 training samples for this task.

Two-view chest X-ray reports tab. 5. Zero-shot; the model was trained on single-view input only. The SOTA column is the single-view SOTA from Table 2.

Positive transfer, 84B with and without CXR classification tab. 6. Single run per variant; no variance is reported.

Radiologist side-by-side fig. 4 §6.3.1. Share of cases in which each candidate was ranked best of four, averaged over four raters: reference report 37.14%; Med-PaLM M 84B 25.78%; 12B 19.49%; 562B 17.59%. Pairwise preference over the reference, averaged over raters: 84B 40.50%; 12B 34.05%; 562B 32.00% §6.3.1. Per-rater breakdown is in fig. 4 (subfigure b).

Radiologist independent evaluation tab. A.8 §6.3.2. Mean per report, with 95% CIs as given in the source.

Language-only QA against PaLM tab. A.3. Med-PaLM M outperforms the PaLM baseline by a large margin on all three QA sets, but stays below Med-PaLM 2 with ensemble refinement. PaLM few-shot results are from [singhal2022large]. PaLM (MedQA 58.90%, MedMCQA 54.50%, PubMedQA 55.00%) is in tab. A.3.

Paper vs code

No official code. The paper releases no code and says the LLMs will not be open-sourced [Code Availability statement after §9]. The paper links no project page. A GitHub repository search for the title and "Med-PaLM M" returned only a third-party analysis repository by a non-author, which was not used.

Pitfalls

• Missing training details. Not stated: number of training steps, warm-up, hardware, decoding strategy, loss form, random seeds, and how the training exemplar is chosen. Only the TB zero-shot exemplar is said to be hand-written §5.2. An implementer must pick these and should record the choice.
• Shot counts and the word "zero-shot". For most tasks the exemplar is text-only with its image replaced by the placeholder text, and each fine-tuning input has at most one image §4.2 fig. 2. Tasks in the mix use 0, 1 or 2 shots tab. A.1. The TB result is called zero-shot, although a text exemplar is shown §5.2.
• Architecture naming. The introduction calls Med-PaLM M a "sequence-to-sequence" model §1, but the PaLM base is a decoder-only Transformer §4.1. Image tokens are interleaved with text tokens in one input sequence §4.2.
• Intro figure vs appendix. The introduction and the radiologist results state 0.25 clinically significant errors per report for the best model §1 §6.3.2. The appendix table shows 0.23 for significant clinical errors and 0.25 for total clinical errors at 84B tab. A.8. The 0.25 figure is the total clinical-error count. The best model also differs by measure: 84B has the lowest clinical errors, while the appendix text calls the model with 0.58 total errors the "best performing" model, which is 12B app. A.4 text [tab. A.8].
• Resolution and token budget. Images are 224 x 224, which the authors say loses resolution. They chose this to keep image tokens within the 710-token input limit app. A.3 text tab. A.2. Fine detail such as small lesions may be lost.
• Variant-calling image. DeepVariant's (100, 221, 6) tensor becomes (200, 221, 3) by stacking channels 1-3 over 4-6, then is padded to 224 x 224 app. A.1.2. The text says "stacked" and gives the output shape. The stacking axis is inferred from the shapes, and the padding placement is not stated.
• Non-comparable SOTA. PAD-UFES-20 has no official split, so the random 80/20 split is not comparable to prior work app. A.3 text. CBIS-DDSM uses a 3-class patch-level task, while most prior work uses 2 classes [app. A.3 text]. VQA is scored with BLEU-1 and token F1 rather than accuracy, so the numbers are not directly comparable to classification-style VQA results app. A.3 text. Some SOTA cells, such as the QA rows, come from Med-PaLM 2 with ensemble refinement, a different system app. A.3 text.
• Leakage risk. The clinical-T5 summarization baseline may have been pretrained on MIMIC-III test data app. A.3 text. The paper does not confirm or rule this out for its own pretraining data, which is PaLM-E.
• MIMIC-CXR sample definition. Each image from a multi-view report is an independent sample, which differs from most prior work that uses only frontal views app. A.1.2. Reported CXR numbers are not directly comparable with frontal-only results.
• Filters that change the task. MIMIC-CXR findings over 800 characters are dropped; MIMIC-III findings over 600 tokens are dropped app. A.1.1 [app. A.1.2]. Results on the filtered sets do not describe the full corpus.
• Verbosity and scaling. The 562B model is more verbose and does not beat 84B on most CXR metrics §6.1 text tab. 3. The authors suggest that retrieval-based approaches may fit this output space better §6.1 text.
• Hallucinated prior references. Reports in MIMIC-CXR refer to prior studies or missing views. The model learns to produce these, and the evaluation counts them as non-clinical errors app. A.4 text tab. A.8. CXR-PRO is suggested for a cleaned set app. A.4 text.
• Human-evaluation limits. Four raters based in India; each side-by-side case is judged by one rater; there is high inter-rater variability §5.3 app. A.4 text. The paper reports only clinically relevant errors in the main analysis §6.3.2.
• Text-overlap metrics. Med-PaLM M does not beat SOTA on BLEU-1, BLEU-4, ROUGE-L or CIDEr-D; the authors argue these metrics miss factual errors app. A.3 text tab. 2.
• Single runs. The positive-transfer ablation and the TB and two-view results come from single runs at one scale §5.2 tab. 5 tab. 6. No seed variance is reported.
• Multi-image inference. The claim that the model handles several images at inference §4.2 is not evaluated beyond the two-view setup tab. 5.
• Figure and table numbers. The numbers here follow the order of the LaTeX source, and the compiled PDF was not checked. Verify figure and table numbers against the PDF before citing them externally.
• Not reproducible as is. PaLM-E checkpoints are required, and the LLMs are not released [Code Availability]. The authors describe the method so that similar approaches can be tried with other LLMs [Code Availability]; the paper does not test any other backbone.

Lineage

• Builds on: PaLM for the language model; PaLM-E (arXiv:2303.03378) for the multimodal base; ViT (arXiv:2010.11929) for the vision encoder; 4B ViT from PaLI (arXiv:2209.06794) and 22B ViT (arXiv:2302.05442). Med-PaLM M adds biomedical fine-tuning on MultiMedBench §2.1 §4.1.
• Compared against: Med-PaLM 2 (arXiv:2305.09617) and Med-PaLM (arXiv:2212.13138) for QA; LLaVA-Med (arXiv:2306.00890) as the closest multimodal biomedical model §2.2; BenchMD as the nearest benchmark §2.3; PaLM-E 84B as the generalist baseline §5.1.
• Superseded by: not known from this card.