Open Foundation Models in Healthcare: Challenges, Paradoxes, and Opportunities with GenAI Driven Personalized Prescription

In plain words

Open models are AI systems whose trained weights anyone can download and run on their own hardware. This paper asks whether such models can help check that a medicine suits a particular patient. The authors wrote 100 made-up patient records based on patterns in a public critical-care database. Each record names one of five common medicines, and the models are asked to flag risks such as drug interactions, wrong doses and pregnancy concerns.

The authors also tried giving each model the official product information for the medicine, found with a text-similarity search, before it answered. Open models did about as well as a commercial model on overall judgement, and the extra text helped most models a little. Human raters did not always agree that it made the answers better. The paper also surveys which healthcare models are truly open, and it points out that "open" often means different things.

TL;DR

• Survey and taxonomy of open-weight LLMs and foundation models for healthcare, with a case study on personalized prescriptions §I §III.
• Four models (Llama 4 Scout, Mistral Small 3.2 24B, MedGemma 27B, GPT-5.6 Luna) label medication suitability for 100 synthetic profiles §IV-B.
• Without retrieval, the three open models score 0.810 to 0.850 on overall suitability against 0.780 for GPT-5.6 Luna, with overlapping intervals §IV-C.
• Retrieval from the product information raises overall accuracy for all four models, while blinded two-rater means drop for two of them §IV-C tab. VII.

Problem

Clinicians must weigh many guidance documents and patient-specific data before prescribing, and manual review is hard in short consultations §IV-A. The authors ask whether open LLMs can judge whether a proposed medicine is suitable for a patient and identify patient-specific risks such as contraindications, interactions and dosing concerns §IV, opening. Three research questions are posed: accuracy against a proprietary baseline, limitations, and whether retrieval-based grounding narrows the gap §IV, opening.

The survey side asks a related question about openness. The paper argues that model weights are necessary for openness but are not sufficient under the Open Source AI Definition v1.0, and it applies that rule to the models it catalogues §II-B §III-B.

Method

The case study has four steps: build synthetic patient profiles from MIMIC-III-informed templates, have a clinician verify them, run the LLMs on the suitability task, and validate the results quantitatively and qualitatively §IV-B fig. 3.

Data. The dataset has 100 synthetic profiles, 20 for each of five medicines: warfarin, metformin, levothyroxine, lisinopril and omeprazole. Each profile is a structured JSON record with demographics, primary diagnosis, proposed medicine and dose, comorbidities, allergies, current medications, pregnancy and lactation status, and one of five predefined risk scenarios. MIMIC-III informed the choice of attributes, but no real patient records are reproduced §IV-B, Data Description. A clinician co-author checked every profile and corrected inconsistencies §IV-B, Verification of Patient Profiles.

Prompt. The model receives a fixed prompt that names it "Charlie", includes the patient profile, and asks for a JSON answer tab. V. For each of eight fields (age, dose, comorbidities, contraindications, pregnancy, lactation, warnings, genetics) the answer is "Suitable", "Risky" or "N/A" with a reason. It also asks for an overall result and a score from 0 to 100 tab. V. The paper does not show where retrieved passages enter the prompt, and the template in tab. V shows only the profile placeholder.

Retrieval (RAG). For the RAG condition, the SmPC of the medicine is split into overlapping word chunks. Chunks are ranked by TF-IDF cosine similarity against the query, and the top-ranked passages are supplied to the model §IV-B, Evaluating LLMs §IV-C, Experimental Setup. Chunk size, overlap and the number of passages are not stated.

Models. Three of the four models were reached through OpenRouter. The fourth, the medical-domain model, ran on a Hugging Face endpoint §IV-C, Experimental Setup. The paper describes the open-weight models as general-purpose (Llama 4 Scout, Mistral Small 3.2) or medical-domain (MedGemma 27B) §IV-B, Evaluating LLMs.

Automatic scoring. Each field is compared with a reference label derived from the SmPC. Accuracy, macro precision, recall and F1 are computed per field and for overall suitability. For pregnancy, lactation and genetics, "N/A" and "Suitable" count as equal §IV-C, Inference Configuration. Mean field accuracy is a plain average of the eight per-field accuracies §IV-C, Inference Configuration.

Blinded review. Two raters, a pharmacy student and a clinician co-author, score a masked sample of 96 items (12 profiles, four models, two retrieval settings) on five dimensions from 0 to 5: Medication Selection Accuracy (MSA), Drug Interaction Detection (DID), Patient-Specific Dosage Adjustment (PSDA), Prescription Safety Score (PSS) and General Assessment (GA) §IV-B, Performance Evaluation §IV-C, Evaluation Strategy. Ratings are independent, response order is randomized and model and condition are concealed §IV-C, Evaluation Strategy. Quadratic-weighted Cohen's kappa is computed per dimension, and the mean of the two raters is used for model-level summaries §IV-B, Performance Evaluation.

# Medication suitability evaluation  [§IV-B] [tab. V]
# Input:  profiles (100 records), smpc_text per medicine
# Output: field labels and overall label per (model, profile, condition)

def evaluate(models, profiles, smpc_text, use_rag):
    # Split the SmPC into overlapping word chunks  [§IV-B, Evaluating LLMs]
    chunks = split_overlapping_words(smpc_text)    # sizes not stated
    records = []
    for profile in profiles:
        passages = []
        if use_rag:
            # Rank chunks by TF-IDF cosine similarity; k not stated  [§IV-C]
            passages = top_ranked_tfidf(chunks, profile, k=K_NOT_STATED)
        prompt = fill_template(profile, passages)  # placement not stated
        for model in models:
            # Temperature 0, at most 1,800 output tokens  [§IV-C]
            reply = model.generate(prompt, temperature=0, max_tokens=1800)
            records.append(parse_json_labels(reply))
    return records

Hyperparameters

Results

Author-reported. Setup: 100 synthetic profiles, one proposed medicine each, GPT-5.6 Luna as the proprietary baseline, first of two runs §IV-C, Quantitative Results.

Full field-level results from tab. VI. P, R and F1 are macro-averaged.

Derived figures, as stated in the text §IV-C, Quantitative Results:

• Mean field accuracy without RAG: GPT-5.6 Luna 0.820, Mistral Small 3.2 0.808 (the paper's own rounding), Llama 4 Scout 0.785, MedGemma 27B 0.754. With RAG: 0.824, 0.811, 0.804 and 0.759 respectively.
• Overall-suitability accuracy: 0.780 (GPT-5.6 Luna), 0.820 (Llama 4 Scout), 0.850 (MedGemma 27B) and 0.810 (Mistral Small 3.2) without RAG. With RAG: 0.930, 0.900, 0.930 and 0.890.
• Binomial 95% Wilson intervals, n = 100, overall suitability §IV-C, Uncertainty: GPT-5.6 Luna 0.780 [0.689, 0.850] without RAG and 0.930 [0.863, 0.966] with RAG; Llama 4 Scout 0.820 [0.733, 0.883] and 0.900 [0.826, 0.945]; MedGemma 27B 0.850 [0.767, 0.907] and 0.930 [0.863, 0.966]; Mistral Small 3.2 0.810 [0.722, 0.875] and 0.890 [0.814, 0.937]. The paper calls between-model ranking descriptive.
• Run-to-run mean field agreement between the two repetitions: 0.926 to 0.950 (GPT-5.6 Luna), 0.944 to 0.950 (Llama 4 Scout), 0.910 (MedGemma 27B), 0.971 to 0.981 (Mistral Small 3.2) §IV-C, Run-to-run stability.

Blinded two-rater review, mean of the two raters, scale 0 to 5, 12 profiles tab. VII:

Agreement between the raters on the 96 items: quadratic-weighted kappa 0.96 (MSA), 0.59 (DID), 0.69 (PSDA), 0.84 (PSS), 0.69 (GA); exact agreement 99%, 82%, 77%, 92% and 83%. No paired scores differed by more than one point. 98% of DID ratings were 4 or 5 §IV-C, Results using Subjective Assessment.

Findings the authors state §IV-C, Quantitative Results §IV-C, Results using Subjective Assessment §V:

• Retrieval improved overall-suitability accuracy for every model. The largest field gains were on dose and contraindications. Comorbidity and warning accuracy declined for several models.
• Retrieval lowered the two-rater means of Llama 4 Scout and MedGemma 27B on all five dimensions. GPT-5.6 Luna rose on all five with RAG.
• The authors do not report a hallucination rate, because the task is a constrained three-way classification against reference labels §IV-C, Uncertainty.

Survey results: the taxonomy covers healthcare models with weight release, license class, training code, training data and OSAID status tab. II. A separate table gives the release openness of 45 models by license class tab. IV. Healthcare datasets are grouped by access tier tab. III. The survey covers 2019 to September 2026 §III-B. The figures are counts from the paper's own scoring, not checked here.

Paper vs code

No official code. The paper's abstract and its data-availability sentence say that data and code "will be released upon publication" [abstract]. The LaTeX has no repository, project page or code link. A GitHub repository search for "personalized prescription SmPC" and for "open foundation models healthcare prescription" returned zero repositories (checked 2026-11-05). No code field is set.

Pitfalls

• Overall accuracy is a three-way label match on one dataset. It is computed on 100 synthetic profiles drawn from five medicines, 20 each. The clinician who checked the profiles is a co-author §IV-B, Verification. The authors say the dataset is for research evaluation, not clinical use §IV-B, Data Description.
• Field accuracy is inflated for three fields. "N/A" and "Suitable" are treated as equal for pregnancy, lactation and genetics §IV-C, Inference Configuration. Genetics macro-F1 is 0.49 to 0.50 for every model and condition in tab. VI except MedGemma 27B without RAG (1.000), so its high accuracy carries little information. The paper says so §IV-C, Quantitative Results.
• Between-model ranking is not established. The paper's own intervals overlap across models within each condition §IV-C, Uncertainty. The abstract's phrase "matched or exceeded" rests on point estimates. Only the GPT-5.6 Luna RAG-versus-no-RAG intervals fail to overlap.
• Retrieval details are missing. Chunk size, chunk overlap, the number of retrieved passages, and where passages enter the prompt are not stated §IV-B, Evaluating LLMs tab. V. RAG results cannot be reproduced exactly from the text. The paper also does not say how the reported "overall score" (0 to 100) maps to the Suitable or Risky label tab. V.
• Model snapshots are unpinned. Three of the four models were reached through OpenRouter and the fourth through a Hugging Face endpoint §IV-C, Experimental Setup. The exact model version, quantisation and hosting provider are not stated. Temperature 0 is set, but no seed is given. The two runs agree closely (0.910 to 0.981 mean field agreement) but are not pooled §IV-C, Run-to-run stability.
• Expert ratings and label accuracy disagree. The authors warn that the two measures are complementary, not interchangeable, and that DID ratings should not stand in for the contraindication field §IV-C, Results using Subjective Assessment §V. Plan for both measures if you use this setup.
• The human review is small. It covers 12 profiles and two raters, and kappa for DID is 0.59 under a ceiling effect §IV-C, Results using Subjective Assessment. The paper calls its ranking of conditions descriptive, given standard deviations of 0.00 to 0.62 §IV-C, Results using Subjective Assessment.
• All data are English. The paper notes this limitation and that the evaluation covers only five medicines §IV-C, Results using Subjective Assessment.
• Numbering. Section, table and figure numbers follow the order in the LaTeX source and the IEEEtran convention. The PDF was not compiled here, so the printed numbers should match but were not checked against it.

Lineage

• Builds on: the medication-direction work of Pais et al., which the authors cite as motivation for the case study §IV intro. The authors contrast it with their focus on whether a medicine is suitable in the first place §IV intro.
• Compared against: a set of related healthcare foundation-model surveys, listed in tab. I, and the proprietary GPT-5.6 Luna baseline tab. VI.
• Superseded by: not known from this card.