BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

In plain words

Large general models can answer medical questions, but the strongest ones are closed, and the open medical models that existed were limited, mostly because few were lightweight and allowed commercial use. The authors took an open general model and kept training it on medical research articles from PubMed Central, the collection of open-access papers, so it picks up more medical vocabulary and facts.

They then gave the result a broad test: ten medical question-answering sets in English, plus machine-translated copies in seven other languages. They also compared smaller, quantized versions of the model, and blended it with its general parent model to see whether that keeps general skill while adding medical skill. The numbers are the authors' own.

TL;DR

• BioMistral 7B is Mistral 7B Instruct v0.1 further pre-trained on 3B tokens of PMC Open Access text (Commercial Use Allowed subset) §3.1, §3.2.
• It is evaluated on 10 English medical QA tasks in 3-shot and after LoRA fine-tuning, and on 7 machine-translated languages §4.1, §4.2, §5.1, §5.2.
• Author-reported: merged variants (SLERP, DARE, TIES) and the Ensemble beat the plain BioMistral 7B on the 3-shot English average; GPT-3.5 Turbo 1106 remains higher (66.0) tab. 2.
• Quantized variants (AWQ 4-bit, BitsandBytes 4/8-bit) are compared on accuracy, memory and speed §5.5, tab. 4, tab. 7.
• Relevant if you want an open medical 7B model; read the licence and reproducibility pitfalls before relying on it.

Problem

Generalist LLMs can answer medical questions, but using proprietary models raises data-privacy concerns. The open medical models before this paper were limited, because few were lightweight and allowed commercial use with performance close to larger or proprietary models §1. The paper's answer is to continue pre-training a general model on domain text §3.2.

Method

The paper has four components: a pre-training corpus and continued pre-training §3.1, §3.2; model merging §3.3; quantization §3.4; and an evaluation protocol with SFT §4. Everything below is the authors' design unless marked as code-only.

3.1 Pre-training corpus

• Source: the PMC Open Access Subset, restricted to its Commercial Use Allowed part, whose documents carry CC0, CC BY, CC BY-SA or CC BY-ND licences §3.1.
• Selection: a 3-billion-token subsample of the processed corpus, about 1.47 million documents. It is 98.75% English; the rest covers nine other languages (Dutch, German, French and others). Non-English documents were prioritised and English was used to fill the target §3.1.
• Epoch target: the authors aimed for 1.5 passes over this subsample, which they took from the Zephyr study as a useful point before returns diminish, and which fit the 20-hour compute limit §3.1, app. E.
• Tokenization: the Mistral SentencePiece tokenizer (32,000 tokens) with one added padding token that is identical to the end-of-sequence token app. B.

3.2 Continued pre-training

• Base model: Mistral 7B Instruct v0.1. Architecture details carried over are grouped-query attention, sliding-window attention and a rolling buffer cache §3.2.
• Context length 2,048 tokens, FlashAttention-2 §3.2.
• Optimizer AdamW with a cosine learning-rate schedule, learning rate 2e-5, no warm-up, weight decay 0.01, gradient accumulation 2, per-GPU batch 16, on 32 A100 80 GB GPUs for a total batch of 1,024; Fully Sharded Data Parallel; BF16 precision §3.2.
• Sequence packing ("post-tokenization grouping"): documents are concatenated, each ending in the end-of-sequence token, and cut into full 2,048-token sequences with no padding. The authors report this cuts the sequence count by 87.88% and shortens epochs §3.2, alg. 1.

# Pack tokenized documents into full 2,048-token rows  [§3.2] [alg. 1]
# Input:  documents, a list of token lists (each ends with </s>)
# Output: rows, a list of token lists of exactly 2,048 tokens

def group_documents(documents, seq_len=2048):
    # Concatenate every document into one long token stream  [alg. 1]
    tokens = flatten(documents)
    length = len(tokens)
    # Keep only whole rows; the short tail is dropped  [alg. 1]
    if length >= seq_len:
        length = (length // seq_len) * seq_len
    # Cut the stream into consecutive rows from index 0  [alg. 1]
    # [code: pretrained/dataset/preprocessing_dataset.py#L56-L70]
    rows = []
    for start in range(0, length, seq_len):
        rows.append(tokens[start : start + seq_len])
    return rows

3.3 Model merging

• Merges combine BioMistral 7B with Mistral 7B Instruct v0.1 without further training §3.3, §5.3.
• SLERP: spherical linear interpolation of the two parameter sets, intended to avoid the information loss of plain weight averaging §3.3.
• TIES: build a "task vector" for each model by subtracting the base model, reduce interference by keeping sparse vectors and using a sign-consensus step, then add the merged vector back to the base §3.3.
• DARE: applied on top of TIES; most delta parameters are set to zero by random pruning and the rest rescaled, which the paper reports keeps or improves performance §3.3.
• The "Ensemble" baseline is not a merged checkpoint. It combines the two models' per-letter scores at inference; the paper calls this log probabilities, the code sums probabilities (see Paper vs code) §5.1, §5.3.

3.4 Quantization

• Activation-aware Weight Quantization (AWQ) keeps the most important weights unquantized, based on the observation that weights differ in importance §3.4.
• BitsandBytes (BnB) quantizes the whole model to a fixed 4 or 8 bits §3.4.
• Memory and speed were measured on the MMLU professional-medicine set on an RTX 3090 app. D, tab. 7.

4.1 Downstream tasks

• MMLU, six medical subjects (clinical knowledge, medical genetics, anatomy, professional medicine, college biology, college medicine), 1,089 questions combined. MMLU has no training split, so the authors fine-tuned on MedQA and tested on MMLU §4.1, tab. 1.
• MedQA: USMLE-style questions, four-option (MedQA) and five-option (MedQA 5 opts) configurations; 10,178 training and 1,273 test questions §4.1, tab. 1.
• MedMCQA: four options from Indian entrance exams. Test answers are not public, so the 4,183-question validation split is the evaluation set; the training set was randomly split into 146k train and 37k validation for hyperparameter tuning §4.1.
• PubMedQA, reasoning-required setting: answer yes, no or maybe for a PubMed abstract and question. Fine-tuning uses 211k artificially labelled samples; evaluation uses 500 expert-labelled validation and 500 test samples §4.1.

4.2 Multilingual evaluation

• The English benchmark was machine-translated into Spanish, German, Portuguese, Russian, French, Arabic and Chinese with GPT-3.5 Turbo 1106. The translation covers questions, options and context; the few-shot examples are unchanged §4.2.

4.3 Instruction prompting

• Every task is multiple choice with letter options (A to D, or A to E for five-option MedQA). The prompt template is shown in fig. 2 and follows the instruction style the authors took from GPT-4's medical evaluation [§4.3, app. F].
• Scoring: the next-token distribution is computed once, and only the option letters are considered. The authors argue this prevents irrelevant tokens and hallucinated answers §4.3.

# Choose an answer letter from one forward pass  [§4.3] [fig. 2]
# Input:  prompt ending with "**Answer:**(", option letters
# Output: the chosen letter

def predict_letter(model, tokenizer, prompt, letters):
    # One forward pass; keep only the next-token distribution  [§4.3]
    input_ids = tokenizer(prompt)
    probs = softmax(model.next_token_logits(input_ids))
    # Score only the option letters  [§4.3]
    # [code: Evaluation/FewShot/PredictFewShot-Avg-Thread.py#L31-L43]
    scores = {letter: probs[token_id(letter)] for letter in letters}
    return max(letters, key=lambda letter: scores[letter])

4.4 Supervised fine-tuning (SFT)

• Method: QLoRA fine-tuning together with 8-bit quantization, chosen as lower-cost alternatives to full fine-tuning, plus the batching change from §3.2 §4.4. Hyperparameters are in tab. 6.

Hyperparameters

Values below come from the paper unless the "Where" column names code. "Not stated" means the paper and the code do not give the value.

Continued pre-training

Supervised fine-tuning

Evaluation, merging, quantization

Dataset statistics

Source: tab. 1. The "MedQA" row covers both configurations.

Results

Author-reported throughout. Setup: 7B models, accuracy (higher is better), English unless stated. Values are means; the per-task standard deviations are in the source tables.

Few-shot, English §5.1, tab. 2

Headline claims, as the paper states them: BioMistral 7B beats Mistral 7B Instruct on 8 of 10 tasks in this table, and beats all open-source biomedical baselines on every task §5.1. The average is pulled down by PubMedQA (37.6 vs 72.2); see Pitfalls.

Supervised fine-tuning, English §5.2, tab. 3

The GPT-3.5 row is copied from the few-shot table, as the paper's footnote says tab. 3.

Quantized BioMistral 7B, 3-shot §5.5, tab. 4

Speed and memory, on an RTX 3090 for the MMLU professional-medicine set app. D, tab. 7:

The paper recommends AWQ + GEMM as the most attractive trade-off §5.5.

Model merging §5.3, tab. 2, tab. 3

• Merged variants beat plain BioMistral 7B on the 3-shot average: SLERP +5.1, DARE +4.3, TIES +0.8 points §5.1, §5.3. The text gives these as 5.11%, 4.35% and 0.82% (relative to the base average).
• In SFT the best merge (DARE, 59.4) is 2.1 points above BioMistral 7B (57.3) and 3.5 above Mistral 7B Instruct (55.9) §5.2, §5.3, tab. 3.

Multilingual, few-shot, average over 10 tasks §5.4, tab. 9

Per-task values are in tab. 9 and app. H. The average column is shown here.

The paper notes that the translations are machine-made and that most language gaps trace to translation quality and the training data mix §5.4.

Calibration, Expected Calibration Error, lower is better §5.6, tab. 5

In this table BioMistral 7B and its base model have higher (worse) ECE than the other medical models. The paper attributes this possibly to calibration baselines of the Llama foundation models, and says merging tends to reduce calibration §5.6. The per-language differences from Mistral 7B Instruct are in tab. 5.

Truthfulness, medical subset of TruthfulQA, zero-shot §5.7, tab. 8

Accuracy averaged over health, nutrition, psychology and science categories tab. 8. The prompts are in fig. 3 and fig. 4.

Paper vs code

Code: https://github.com/BioMistral/BioMistral at commit 4d1fa85e271389cc4c0055e9ffa383d592d73e02, linked from the paper's footnote. The repository has no LICENSE file; we cite it but do not quote it.

1. Per-GPU batch size. The paper gives a per-GPU batch of 16, with 2 accumulation steps on 32 GPUs, for a total of 1,024 §3.2. The run script passes --batch_size=32 to a trainer that uses 2 accumulation steps on 32 tasks, which gives 2,048 [code: pretrained/scripts/run_training_PubMed.sh#L4, #L39] training_mistral_7B-LM.py.
2. Pre-training epochs. The paper targets 1.5 epochs §3.1. The run script passes --epochs=3 run_training_PubMed.sh. Which run produced the released weights is not stated.
3. Padding token. The paper says the padding token is identical to the end-of-sequence token app. B. The code adds a separate <pad> special token training_mistral_7B-LM.py preprocessing_dataset.py.
4. Grouping algorithm. Alg. 1 loops over indices starting at 2,048 and assigns each slice to result, which does not match a clean list of rows alg. 1. The code starts at 0, appends every row, and drops the short tail preprocessing_dataset.py. It also keeps only rows of full length preprocessing_dataset.py.
5. Settings the paper omits. The code enables gradient checkpointing, clips gradient norm at 1.0 and turns on tf32 [code: pretrained/scripts/training_mistral_7B-LM.py#L50, #L66, #L74, #L120]. It appends the end-of-sequence token to every document before grouping [code: pretrained/dataset/preprocessing_dataset.py#L36-L38].
6. SFT epochs and GPUs. tab. 6 gives 3 epochs and 8 GPUs. The launch script in the repo uses one GPU and --epochs=6 [code: sft/run_SFT_Mistral.sh#L4-L6, #L39]. The LoRA settings match training_Mistral-SFT_multigpu.py.
7. Merging weights. The paper states equal 50/50 weights §5.3. The SLERP configs interpolate the t value across layers, with separate schedules for attention and MLP blocks and a default of 0.5 MergeSLERP.py. The repo's SLERP script also builds variants merged with Starling, OpenHermes and Zephyr, which are not described in the paper MergeSLERP.py.
8. DARE method. The paper describes DARE as random pruning plus rescaling §3.3. The config uses mergekit's dare_ties method, which adds TIES-style sign handling MergeDARE.py.
9. Ensemble score. The paper says the ensemble "aggregates log probabilities" §5.3. The code adds the per-letter softmax probabilities [code: Evaluation/FewShot/PredictFewShot-Avg-Thread.py#L31-L35] [code: Evaluation/FewShot/EvaluateFewShot-Ensemble.py#L95].
10. AWQ group size. The paper does not give one §3.4. The code uses 4-bit weights, group size 128, zero point, and GEMV as the default kernel QuantToAWQ.py.
11. Few-shot prompts. The prompts are loaded from an external dataset, Project44/EnglishOnlyBioInstructQA, not from the repo [code: Evaluation/FewShot/PredictFewShot-Avg-Thread.py#L73]. The exact in-context examples cannot be checked from this code.
12. ECE bins. The paper gives no bin count §5.6. The code uses 5 bins StudyCalibration-Langs.py.
13. Not reviewed. The corpus-building script Data/Training/PubMed.py was not read in detail, so the license filter for the corpus is not verified against code.

Pitfalls

• Batch and epoch numbers disagree. Batch size, epoch count and GPU count differ between the paper text, the tables and the scripts (see Paper vs code, items 1, 2 and 6). Pick one set and say which you used.
• Grouping off-by-one-block. Following Alg. 1 literally drops the first 2,048-token block. The code's version starts at 0. Also, the tail is dropped, which is up to 2,047 tokens per run.
• Padding token. Setting the pad token equal to , as the paper says, and adding a new <pad> token, as the code does, give different token IDs and vocabulary sizes. Decide which you use before loading weights.
• Ensemble aggregation. The paper's ensemble is described in log space; the code adds probabilities. These are different aggregations, so do not assume a result from one carries over to the other.
• Average hides PubMedQA. BioMistral 7B's 3-shot average (50.3) is below Mistral 7B Instruct's (51.2) because PubMedQA is 37.6 vs 72.2. Per task, BioMistral wins 8 of 10 tab. 2.
• Text and table mismatch on quantization. The text gives BnB 8-bit gains of 0.65 (Clinical KG) and 1.00 (Anatomy) §5.5. Table 4 shows 61.3 vs 60.9 (+0.4) and 50.1 vs 49.6 (+0.5). Use the table.
• Text and table mismatch on TruthfulQA. The text says BioMistral 7B outperforms all models on both prompts and beats GPT-3.5 Turbo by 4.0% §5.7. Table 8 shows the Ensemble higher on both prompts (57.6, 65.6), GPT-3.5 Turbo higher on Prompt 1 (53.6 vs 51.6), and only a 2.6 point lead on Prompt 2.
• Quantized average above the baseline. AWQ 4-bit scores 51.8 against 50.3 for FP16. The gain comes from PubMedQA (+24.1 points); the other nine tasks lose about 1.1 points on average tab. 4.
• Speed trade-off. AWQ + GEMV takes about 10 times longer than FP16 (421.78 s vs 40.94 s) on the same test tab. 7. Choose the kernel with this in mind.
• Translation circularity. GPT-3.5 Turbo 1106 produced the translations and is also one of the benchmarked models §4.2. The paper reports no human check of the translations, so treat cross-lingual numbers as relative only.
• Ambiguous SFT caption. The SFT table caption says results are "averaged across 3 random seeds of 3-shot" tab. 3, which does not describe fine-tuning.
• MedMCQA evaluation split. Test answers are unavailable, so results use the validation split §4.1. Compare numbers only to other papers that used the same split.
• MMLU under SFT. MMLU has no training set, so the SFT numbers for it come from models fine-tuned on MedQA §4.1. This is transfer, not in-domain training.
• Compute and cost. The authors report about 5,000 A100 80 GB GPU hours and 355.47 USD of GPT-3.5 Turbo inference for translation and evaluation [Limitations]. Reproducing the full study is expensive.
• Data licences. The pre-training corpus includes CC BY-ND documents §3.1. The paper states the models and datasets are released under Apache 2.0 [§1, footnote]. The repo has no LICENSE file, so do not assume the code licence. Check the model card before redistributing weights.
• Use limits. The authors say the models are for research use and advise against deployment for clinical or other professional use without further evaluation [Ethics Statement]. Evaluation covered MCQA only [Ethics Statement].

Lineage

• Builds on: Mistral 7B Instruct v0.1 §3.2; Zephyr, for the 1.5-epoch target §3.1; FlashAttention-2 §3.2; QLoRA §4.4; AWQ §3.4; TIES, DARE and SLERP for merging §3.3.
• Compared against: MedAlpaca 7B, PMC-LLaMA 7B, MediTron-7B, BioMedGPT-LM-7B, and GPT-3.5 Turbo 1106 §5.1, tab. 2.
• Superseded by: not stated in this paper.