Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data

In plain words

Radiologists read X-rays, CT and MRI scans, often several at a time and next to a patient's history. Most earlier AI models for medical images handled only flat 2D pictures, one picture per question, and were built for a single task. RadFM is one model that accepts text mixed with one or more scans, flat or 3D, and writes the answer: it can name the scan type, suggest a diagnosis, answer a question, write a report or explain its reasoning.

The authors gathered about 16 million image-text pairs to train it, then trained further on a cleaner radiology subset of about 3 million pairs. They also built a benchmark of five tasks to test it, and had radiologists rate the written answers, because automatic scores can miss clinical accuracy.

TL;DR

• RadFM is a generative vision-language model for radiology. Each 2D or 3D scan is encoded by a 3D ViT, compressed to 32 tokens by a Perceiver, and placed inside a LLaMA-family decoder §3.2.
• Training has two stages: pre-training on MedMD (16M pairs), then instruction tuning on RadMD (3M pairs) §2.1, §3.3.
• Author-reported: on RadBench's five tasks, zero-shot, RadFM scores highest on every metric of the overview table tab. 3.
• The released training script does not reproduce the paper's pre-training stage, and the weighted loss from eq. 2 is computed but not optimised. Read "Paper vs code" before reusing the code.

Problem

• Most medical multimodal foundation models accept only 2D images and a single image per query, which fails for CT and MRI and for longitudinal follow-up §6.1, §6.3.
• Available medical image-text data is mostly 2D figures and captions from papers, or chest X-ray reports. The authors argue that this limits generalisation across modalities and anatomies, and that figure-derived reports are weaker than clinical reports §2.1.
• Medical tasks are usually handled one architecture per task; the authors aim for one generative model that takes any mix of text and scans §1, §6.2.
• No benchmark covered 2D and 3D radiology across several tasks with human rating §1, §2.2, §4.1.3.

Method

Data

MedMD has 16M image-text pairs: about 15.5M 2D images and 500k 3D scans, pooled from PMC-Inline, PMC-OA, PMC-VQA, PMC-CaseReport, MPx, RP3D and public VQA and chest X-ray sets §2.1 tab. 1.

• PMC-Inline (11M, interleaved figures and text from PubMed Central papers) is used only in pre-training §2.1.1 tab. 1.
• PMC-CaseReport: about 103K case reports, with 1.1M question-answer pairs generated with ChatGPT; the patient background is kept as context, and 438k pairs remain after the radiology filter §2.1.2 tab. 1.
• RP3D: 3D radiology cases from Radiopaedia, converted into caption, modality, rationale and VQA formats §2.1.2.

RadMD (about 3M pairs) is the radiology subset used for fine-tuning. The filter removes non-radiology images, all PMC-OA and PMC-Inline data, most 2D PMC-series cases, age and size questions, and balances normal and abnormal diagnosis cases §2.1.3.

RadBench is the evaluation suite: 13 datasets over five tasks §2.2 tab. 2. Test sizes are 14,419 (modality), 112,256 (disease diagnosis, text value), 7,329 (VQA), 2,098 (report generation) and 1,000 (rationale diagnosis) §2.2. Eight annotators checked the VQA, report and rationale cases by hand §2.2. PMC-derived data is split by paper and RP3D by publication date (before 2023 for training) tab. 2 caption.

Formulation

Each sample is a text sequence T with image placeholders and a set of scans V = {v_1, ..., v_N}, where each scan is 2D or 3D §3.1. The model gives the likelihood of the text conditioned on the scans and the preceding text eq. 1. Training minimises a weighted negative log-likelihood eq. 2:

\mathcal{L}_{\text{reg}} = -\sum_l w_l \log \Phi_{\text{RadFM}}(\mathcal{T}_l \mid \mathcal{V}_{<l}, \mathcal{T}_{<l}) \quad [\text{eq. 2}]

The per-token weight depends on the data type:

• Interleaved samples (PMC-Inline): weight 3 for medical (UMLS) words, 1 for other tokens, 0 for image placeholders eq. 3.
• Instruction-response samples: weight 0 for instruction tokens and image placeholders, 3 for response tokens that are UMLS words, 1 for other response tokens eq. 4 §3.1.

Prompts per task §3.1: modality recognition uses yes/no prompts ("Is this image shot by {modality}?") and open prompts; diagnosis uses yes/no prompts ("Does the patient have {disease}?") with balanced classes, and open prompts whose answer is the disease name(s); VQA uses the dataset question and answer; report generation uses prompts such as "What can you find from the scans?"; rationale diagnosis asks for the disease and its typical radiological features. The prompt files are in the repository yes_no_prompt.json radiology_feature_prompt.json.

Architecture

• Visual encoder: a 3D ViT with 12 layers and width 768, learnable 3D positional embeddings, and a 32 × 32 × 4 patch (height × width × depth) §3.2 §3.3.1 my_embedding_layer.py vit_3d.py.
• 2D images are expanded to depth 4 by replicating slices, so 2D and 3D share one encoder §3.2.
• Each scan gives P_i tokens, where P_i depends on depth eq. 5.
• Perceiver: a fixed set of P = 32 learnable queries aggregates the tokens of each scan, so every scan yields 32 tokens regardless of size eq. 6 §3.2.
• Fusion: each image placeholder is replaced by the 32 visual embeddings of its scan; the sequence goes into a decoder-only LLM, which outputs next-token probabilities eq. 7 §3.2. The and tokens are added around each visual block §3.3.1.
• The model has 14B parameters in total §3.3.1.

# Visual encoding and fusion for one sample  [§3.2] [eq. 5-7]
# text_ids: (L,) token ids; each image has a block of 32 placeholder ids
# scans: list of S volumes, each (H, W, D) after preprocessing  [§3.3.1]

def encode_scan(scan):
    # Split the scan into 3D patches and run the 3D ViT  [§3.2] [eq. 5]
    # The number of patch tokens P depends on the scan size
    return vit_3d(scan)                              # (P, 768)

def compress(tokens):
    # Perceiver: 32 learned queries attend to the patch tokens  [§3.2] [eq. 6]
    # The output length is always 32, whatever P was
    return perceiver(tokens)                         # (32, 768)

def forward(text_ids, scans):
    # Encode and compress each scan on its own  [§3.2]
    visual = [compress(encode_scan(s)) for s in scans]
    # Map the visual tokens to the LLM width  [§3.3.1]
    visual = [project(v) for v in visual]            # (32, 5120) each
    # Replace each 32-id placeholder block by its scan's 32 embeddings  [eq. 7]
    # The sequence length stays L, because placeholders are replaced one by one
    embeddings = replace_placeholders(text_ids, visual)   # (L, 5120)
    # Decoder-only LLM gives next-token logits  [eq. 7]
    return llm(embeddings)                           # (L, vocab)

Training

• Stage 1: pre-training on all of MedMD for 4 epochs. Stage 2: fine-tuning on RadMD for 4 epochs. All settings are the same in both stages, apart from the data §3.3 §3.3.1.
• The language model is frozen in the first epoch, so the image embeddings can align with the text embeddings; afterwards all parameters are trained §3.3.1.
• Preprocessing: min-max normalisation of intensities; 2D images turned into 3D by adding a depth axis; height and width resized to 512 × 512 for 2D and 256 × 256 for 3D; depth resized to the nearest multiple of 4, not above 64 §3.3.1.
• Maximum sequence length 2048 tokens §3.3.1.

# Preprocess one scan before the 3D ViT  [§3.3.1]
# volume: (H, W, D); 2D images arrive with D = 4  [§3.2]
def preprocess(volume, is_2d):
    # Min-max normalise the intensities; target range not stated  [§3.3.1]
    volume = (volume - volume.min()) / (volume.max() - volume.min())
    # Height and width: 512 for 2D images, 256 for 3D volumes  [§3.3.1]
    side = 512 if is_2d else 256
    # Depth: nearest multiple of 4, at most 64  [§3.3.1]
    depth = min(64, round_to_multiple(volume.shape[2], 4))
    return resize(volume, (side, side, depth))

# Loss weights per token  [eq. 3] [eq. 4]
# Interleaved text has no instruction part: every non-placeholder token counts
def token_weight(is_placeholder, is_instruction, is_umls_word):
    # Image placeholder tokens are never scored  [eq. 3] [eq. 4]
    if is_placeholder:
        return 0
    # Instruction tokens are not scored in instruction-response samples  [eq. 4]
    if is_instruction:
        return 0
    # UMLS medical words get weight 3, other tokens weight 1  [eq. 3] [eq. 4]
    return 3 if is_umls_word else 1


# Weighted negative log-likelihood  [eq. 2]
def weighted_nll(log_probs, targets, weights):
    # log_probs: (L, V) log-probabilities of the next token
    # Sum (not average) of weight times negative log-likelihood  [eq. 2]
    per_token = -log_probs[arange(L), targets]       # (L,)
    return sum(weights * per_token)                  # scalar

Evaluation

• Classification tasks (modality, diagnosis): accuracy and F1 §4.1.2.
• Open-ended tasks (VQA, report, rationale): BLEU and ROUGE with 1-grams by default, and BERT-sim, the F1 of BERTScore §4.1.2 tab. 3 caption.
• UMLS_Precision and UMLS_Recall: the overlap of UMLS medical terms between the prediction and the reference; precision divides the overlap by the terms in the prediction, recall by the terms in the reference §4.1.2.
• Human rating: three radiologists with at least one year of clinical experience score answers from 0 to 5 (garbled to completely correct) §4.1.3. The set is 400 random RP3D test cases per open-ended task, 1.2K in total §4.1.3.
• Baselines: OpenFlamingo, MedVInT and Med-Flamingo, zero-shot and (for the first and third) few-shot; GPT-4V only in human rating §4.1.4. Baselines take one 2D image, so for 3D volumes they get the central slice; radiologists picked up to 4 slices for GPT-4V §4.1.4 §5.1.6.

Hyperparameters

Results

Author-reported. Setup: zero-shot prompting of RadFM and three public baselines on RadBench (test sets from tab. 2); metrics pooled over all test samples of each task; 95% CI in parentheses tab. 3.

Per-dataset zero-shot results are in tab. 4 and tab. 5. A few rows where RadFM is not the top score: on PMC-VQA, MedVInT is higher on BLEU, ROUGE and UMLS_Recall tab. 4; on VinDr-PCXR, MedVInT is higher on F1 (66.29 vs 37.53) tab. 4. Few-shot baselines are in tab. 6 and tab. 7; a few-shot baseline beats zero-shot RadFM on some rows, e.g. MIMIC-CXR BLEU (23.79 for OpenFlamingo few-shot vs 19.43) tab. 7 and VinDr-Mammo F1 (64.92 for Med-Flamingo few-shot vs 62.11) tab. 6.

Human rating (VQA, report, rationale; 1.2K cases, three radiologists, 0 to 5 scale): RadFM scores higher than the other models on all three generative tasks §5.1.6 fig. 9. The text gives an average VQA score of 2.87 for RadFM and 2.13 for GPT-4V §6.5. For GPT-4V, 22 cases were excluded because the model refused them fig. 9 caption.

Transfer after fine-tuning (RadFM used as an initialisation; all values from tab. 8, 95% CI in parentheses; "SOTA" is the value cited in the paper, with its source):

PadChest (zero-shot, 163 classes unseen in training): RadFM's per-class accuracy is only plotted, for the top 100 classes fig. 10; the text gives no single number. The other models score about 50%, i.e. chance §5.3.

Paper vs code

Code: https://github.com/chaoyi-wu/RadFM at commit 8d798c554ca0f40cd4a409f0335cc8512cc9d7cf, MIT licence, owned by the first author's account LICENSE. Discrepancies, each with both citations:

• Pre-training is not in the released training code. The paper pre-trains on MedMD, which includes PMC-Inline and PMC-OA §3.3. In multi_dataset.py the PMC-Inline and PMC-OA loaders are commented out, and the active sources are instruction and VQA sets multi_dataset.py multi_dataset.py. The README says train.py covers pre-training too, but the code does not show it.
• The optimised loss is not the weighted loss of eq. 2. The code computes a weighted loss with torch.sum(w * loss) / torch.sum(w), a normalised average, not the unnormalised sum of eq. 2 eq. 2 multimodality_model.py. The forward pass then returns loss=output['loss'], the language model's own unweighted loss multimodality_model.py, and the trainer optimises outputs["loss"] trainer.py. The weights therefore have no effect on training. Pitfalls below expand this.
• Matching loss. The forward pass also computes a keyword-matching loss and mixes it in as 0.8 · loss + 0.2 · loss_match multimodality_model.py. That combined value is never returned, so it does not affect training. The README says this path is inactive ("always equals zero") README.md. The paper does not describe it.
• Perceiver latent width. The paper says the 6-layer Perceiver has a learnable latent array of 32 × 5120 §3.3.1. The code creates 32 latents of width 768 and a 6-layer resampler helpers.py my_embedding_layer.py, then projects to 5120 with a separate linear layer my_embedding_layer.py my_embedding_layer.py.
• Scan size depends on the batch. The paper fixes 512 × 512 for 2D and 256 × 256 for 3D §3.3.1. The collator sets 512 × 512 by default and uses 256 × 256 only when a single-sample batch has depth above 4 train.py. The dataset loader already resizes every scan to 512 × 512 before that multi_dataset.py multi_dataset.py, so a 3D single-sample batch is resampled twice.
• Depth cap. The paper caps depth at 64 §3.3.1. The collator also caps single-sample batches at 32 when depth exceeds 6 train.py.
• Resize operator. The paper says torchvision.transforms.Resize §3.3.1. The code uses torch.nn.functional.interpolate with no mode argument, which is nearest-neighbour in PyTorch's default train.py multi_dataset.py.
• Freezing. The paper freezes the language model for the first epoch §3.3.1. Neither train.py nor multimodality_model.py sets requires_grad or calls a freeze; the lock() helper in blocks.py is never called from the training path.
• Gradient accumulation and precision. The paper uses 4-step accumulation and FSDP with AMP §3.3.1. The code sets batch sizes 4 (2D) and 1 (3D) train.py but no accumulation, FSDP or AMP settings. Gradient checkpointing is enabled multimodality_model.py.
• Initialisation checkpoint. The paper initialises from MedLLaMA-13B §3.3.1. The code default is a local path, .../Book_mix_2048_13B_full/checkpoint-45800 train.py. The repository does not show that this checkpoint is MedLLaMA.
• Loss weights. The paper weights UMLS words by 3 eq. 3 eq. 4. The code gives 3 to tokens that match the tokenised UMLS terms in the answer, and 0 to masked labels, with 1 otherwise multi_dataset.py multi_dataset.py. The weights are computed but not optimised (see the second item above).
• Label masks. Consistent with the paper: question tokens, padding and image placeholders (ids ≥ 32000) are masked multi_dataset.py eq. 4. The code also writes an end-of-sequence token after the answer and keeps it in the labels multi_dataset.py. The paper does not mention an end-of-sequence token.
• Generation. The paper does not state a decoding setup. The code calls generate with max_new_tokens=200 and top_k=50 multimodality_model.py. Top-k only applies when sampling is enabled; the call does not enable it, so decoding is greedy unless the model's generation config samples.
• Metrics. BLEU, ROUGE, UMLS precision and recall, and BERT-sim are not computed in the repository. test.py only writes the generated outputs to a CSV test.py.

Pitfalls

• Do not assume the paper's loss. Eq. 2 weights tokens, but the released code optimises an unweighted language-model loss. If you need the paper's objective, apply the weights yourself and normalise as you see fit: eq. 2 is an unnormalised sum, while the code divides by the sum of weights.
• Pre-training data is largely absent. PMC-Inline and PMC-OA are not loaded by the released code multi_dataset.py. Rad3D-series and MPx-series are restricted or need a request [data availability table] README.md. Many other sources need credentialed access [data availability table]. Full reproduction of the 16M-pair stage is not possible from the repository alone.
• Unstated optimiser settings. The paper gives no learning rate, schedule, warm-up, weight decay or optimiser betas. The code leaves them to Transformers defaults train.py. Do not copy values from other papers into this card's setting.
• Fixed 32 tokens per scan. A flat X-ray and a 64-slice CT both become 32 tokens eq. 6 §3.2. This compression may limit small findings (inference from the design; the paper does not measure it).
• Placeholder id layout must be preserved. The tokenizer reserves 100 images × 32 placeholder ids after <image> and </image> multi_dataset.py. Each id selects one visual row in the embedding matrix by index my_embedding_layer.py. A port that renumbers tokens breaks the mapping. Scans beyond the 100th in a sample get no placeholder; the loader prints an error multi_dataset.py multi_dataset.py.
• Min-max scope. The paper says "min-max normalization of all images" §3.3.1. It does not say whether each scan is normalised separately or the whole batch, or what range is used.
• Depth rounding. "Nearest multiple of 4" is ambiguous at the midpoints, and the code's cap of 32 for single-sample batches differs from the paper's 64 train.py.
• Paper's numbers do not agree with each other.
  • OpenFlamingo's modality accuracy is 49.47% in the text §5.1.1, 49.41% in tab. 3 and 49.91% in tab. 4.
  • The text says MedVInT is the second-best VQA model and gives UMLS_Precision as 14.55% → 31.77% §5.1.3, but 14.55% is OpenFlamingo in tab. 3; MedVInT's value there is 13.09%.
  • The rationale margins in the text (8.56, 15.46, 23.16 and 10.96 points) do not match tab. 3 against any baseline. For BLEU, 34.60 − 8.40 = 26.20 points.
  • The disease-diagnosis count is 112,256 in the text §2.2, but the rows of tab. 2 sum to 112,436.
  • The RadFM F1 confidence interval in tab. 3 has an upper bound equal to its point estimate (79.62, 80.10).
  • The text says the fine-tuned model "can improve ... text generation quality" §5.2, but tab. 8 shows lower IU-Xray BLEU-1 and BLEU-4 than the cited SOTA.
• Comparisons are not like-for-like. Baselines get the central slice of a 3D volume, GPT-4V gets radiologist-picked slices (at most four), and RadFM gets the full volume §4.1.4 §5.1.6. Few-shot baselines are compared with zero-shot RadFM tab. 6 tab. 7.
• Automatic metrics are weak by the authors' own account. The paper says BLEU, ROUGE and BERTScore miss clinical accuracy, and that UMLS scores "do not fully reflect true performance" §4.1.2 §6.7. Rely on the human rating for clinical claims. The UMLS matching rule (case, abbreviations, synonyms) is not stated.
• Splits. BraTS2019 was re-split 7:3 by the authors, so its numbers are not comparable with published SOTA §4.2. PMC-VQA's test set was cleaned manually and the MedVInT checkpoint re-tested on the cleaned set §4.2.
• Compute and inference. The model has 14B parameters and was trained on 32 A100 80GB GPUs §3.3.1. The README recommends at least one A100 80GB for inference README.md.
• Weights licence not checked. The code is MIT-licensed. The checkpoint is on Hugging Face (linked from the README) and its licence was not checked for this card.

Lineage

• Builds on: LLaMA as the decoder family; MedLLaMA-13B, from PMC-LLaMA (arXiv:2304.14454, no card yet) §3.3.1; the Perceiver (not in the wiki) §3.2; the interleaved MMC4 data format used by Flamingo (not in the wiki) §2.1.1.
• Compared against: OpenFlamingo, MedVInT (PMC-VQA), Med-Flamingo (arXiv:2307.15189, no card yet), GPT-4V §4.1.4.
• Superseded by: none known from the wiki's content.