Towards Building Multilingual Language Model for Medicine

In plain words

Open-source language models know a fair amount of medicine in English, but they do much worse on medical questions in other languages. The authors set out to close that gap for six languages: English, Chinese, Japanese, French, Russian and Spanish.

They did two things. First, they gathered a large pile of medical text in those languages from several places: general web text filtered down to the medical parts, scanned medical textbooks, medical websites and some existing small collections. They then kept training existing models on this text, so the models absorb the medical vocabulary and facts. Second, they built a multiple-choice medical quiz in the same six languages, and added a written explanation for each correct answer, produced by GPT-4 and spot-checked by people.

Models that kept training on the medical text scored higher on the quiz than the models they started from. The best of them, built on Llama 3 8B, also beat its base model on English medical benchmarks. The corpus, the quiz and the models are meant as a shared starting point for other people working on medical language models in more than one language.

TL;DR

• MMedC is a 25.5B-token medical corpus in six languages, built from four source types §4.1 app. C.2.
• MMedBench is 53,566 multiple-choice QA pairs in six languages with GPT-4 rationales; 45,048 train and 8,518 test pairs §4.3.
• Continued auto-regressive training on MMedC, then fine-tuning, lifts the same base model (author-reported). MMed-Llama 3 8B reaches 67.75 average accuracy under full fine-tuning, against 62.79 for Llama 3 8B tab. 1.
• The corpus and benchmark share a six-language scope, so the results say little about languages outside it.
• Code is an official repository, but it covers the InternLM and InternLM 2 runs only; the Llama 3 training recipe is not in it.

Problem

Medical LLMs at the time were mostly English-only, and multilingual general models were weak on medical questions in non-English languages §1. Two gaps are named: no large multilingual medical pre-training corpus, and no multilingual medical benchmark that tests both multiple-choice accuracy and the ability to explain an answer §1. The paper argues that accuracy alone does not test reasoning or long-form generation, and that automatic metrics for free-text rationales are hard to trust §1 §4.3.

Method

Section numbers follow the compiled paper: §1 Introduction, §2 Results, §3 Discussion, §4 Methodology (§4.1 corpus, §4.2 auto-regressive training, §4.3 benchmark, §4.4 English benchmark, §4.5 implementation), §5 Conclusion, §6 Data Availability, §7 Code Availability. Appendix sections use letters (app. A to F). Equation numbers count the four numbered equations in §4 in source order.

MMedC corpus (§4.1). Four sources are combined.

1. Filtered general web text. Starts from the CulturaX multilingual CommonCrawl version §4.1. A text is kept when its medical keyword count and keyword density both pass per-language thresholds. Keyword density is the summed length of keyword occurrences divided by the text length §4.1. The keyword list has 200 terms per language §4.1. Keyword density is an unnumbered formula in §4.1; the decision rule is alg. 1.
2. Medical textbooks. More than 20,000 books were collected, following the PMC-LLaMA approach §4.1. Pages are read with PaddleOCR; text boxes are ordered left-to-right and top-to-bottom. Covers, tables of contents and epilogues are removed by page number §4.1.
3. Medical websites. Medical encyclopedias, consultation and popular-science sites, and medical news sites were crawled as a supplement, because filtered CommonCrawl text cannot be traced back to specific sites §4.1.
4. Existing small corpora. Wikipedia and Baidu Baike (filtered with the same keyword method) and the UFAL Medical Corpus (used directly) §4.1.

The quality check for the filter: 100 random sentences per language were inspected, and on average 98 were medical §4.1.

Next-token training (§4.2). The loss is the negative sum of log-probabilities of each token given its prefix eq. 1. Training is continued from a base model on MMedC. The paper says the objective matches GPT-style next-token prediction §4.2.

MMedBench benchmark (§4.3). Multiple-choice items are collected per language from five QA sets: MedQA (English and Simplified Chinese), lgakuQA (Japanese), FrenchMedMCQA (French), RuMedDaNet (Russian, turned into binary choice) and Head-QA (Spanish) §4.3. Datasets without an official split are split 8:1:1 §4.3. GPT-4 writes a rationale of about 100 words for each item given the question, options and correct answer §4.3. Human checkers judged 200 test items per language; 94.7% of the rationales passed §4.3. This yields 1,136 human-checked rationale items §4.3. GPT-4 also assigns each item a topic from 21 medical subjects, retrying up to four times §4.3 fig. 4.

Evaluation settings (§4.3). Zero-shot prompting of closed API models (GPT-3.5, GPT-4, Gemini 1.0 Pro), and for open models two fine-tuning modes: parameter-efficient LoRA and full fine-tuning §4.3. Open models were fine-tuned on the MMedBench training split tab. 1.

Metrics (§4.3). Accuracy is the share of correct option indices. Rationale quality is scored by BLEU eq. 2, ROUGE eq. 3 and BERTScore recall and F1 eq. 4 (multilingual BERT, no baseline rescaling) §4.3. Human and GPT-4 rankings order the models per item; the best rank gets 6 and the worst gets 1 §4.3. Rationale ranking covered six representative models and 50 cases per language §4.3.

English benchmark (§4.4). Four English sets are reported: MedQA, PubMedQA (PQA-L test split), MedMCQA (official test split) and six MMLU medical subjects, 1,089 questions in total §4.4. Scores for other models are taken from the original papers or a public leaderboard, not re-run tab. 3.

# Medical-text filter  [alg. 1] [§4.1]
# Input:  text (str), keywords (set), lang (str), min_hits, min_density
# Output: bool

def is_medical(text, keywords, lang, min_hits, min_density):
    # Word-segment space-delimited languages; match directly otherwise  [§4.1]
    words = segment(text) if lang.is_space_delimited else text
    # Medical keyword count: distinct keywords found  [§4.1] [alg. 1]
    unique_hits = [k for k in keywords if k in words]
    # Density: keyword characters over text length  [§4.1]
    covered = sum(len(k) * text.count(k) for k in unique_hits)
    density = covered / len(text)
    # Paper rule is strict '>'; the code uses '>='  [alg. 1]
    return len(unique_hits) > min_hits and density > min_density

# Chunk a token stream for auto-regressive training  [§4.5]
# Input:  tokens (list of int), max_len = 2048, overlap = 512
# Output: list of chunks, each (max_len,) tokens

def make_chunks(tokens, max_len=2048, overlap=512):
    # Stride is max_len minus overlap: 1536 tokens  [§4.5]
    stride = max_len - overlap
    # Short streams are padded to one full chunk
    # [code: pretrain/prepare_tokenized_dataset.py#L31-L32]
    if len(tokens) < max_len:
        return [tokens + [pad] * (max_len - len(tokens))]
    chunks = []
    for start in range(0, len(tokens), stride):
        # The last window is aligned to the end of the stream
        # [code: pretrain/prepare_tokenized_dataset.py#L35-L41]
        if start + max_len >= len(tokens):
            chunks.append(tokens[-max_len:])
            break
        chunks.append(tokens[start:start + max_len])
    return chunks

Hyperparameters

Results

Author-reported. Setup: MMedBench multiple-choice accuracy, averaged over six languages tab. 1; model sizes as listed in tab. 1. Numbers are read from the source tables. The paper ranks systems with several metrics, and the human-ranking figure fig. 5 was not transcribed here.

Other headline numbers from the same tables: MMedLM 2 (InternLM 2 base, 7B) reaches 67.30 with full fine-tuning, the InternLM 2 base 58.59 tab. 1. In the English table, GPT-4 averages 85.36 and Med-PaLM 2 averages 86.66, both taken from literature or the leaderboard tab. 3. The rationale numbers are BLEU-1 and ROUGE-1 averages tab. 2.

Open-model zero-shot results are not in the rendered tables; the source keeps them commented out.

Paper vs code

Official code: github.com/MAGIC-AI4Med/MMedLM at 9722bb7, linked in the paper's Code Availability section §7. Searched the paper, abstract and project pages only; no other repository was used. The repository's LICENSE file reads "CC BY-SA" with no version LICENSE. No verbatim code is quoted here.

• Auto-regressive training length. Paper: 20k iterations for the 7B model §4.5. Code: num_train_epochs 3 MMedLM_pretraining.sh. The paper does not convert 20k iterations to epochs, so the two cannot be reconciled from the text.
• Auto-regressive batch and LR. Global batch 512 and LR 2e-5 match the paper for InternLM MMedLM_pretraining.sh. The InternLM 2 script uses the same values MMedLM2_pretraining.sh.
• Auto-regressive schedule. Paper gives no warmup, weight decay or clipping. Code: cosine, warmup 0.01, weight decay 0.01, clip 1.0 MMedLM_pretraining.sh MMedLM_pretraining.sh.
• Full fine-tuning. Paper: LR 1e-6, global batch 128 §4.5. Code matches fullmodel_finetuning.sh. Code adds 5 epochs and warmup 0.03, which the paper does not state.
• LoRA. Paper says LoRA used "a similar training setting as full fine-tuning" §4.5, which implies LR 1e-6. The script uses LR 2e-5 and 3 epochs LoRA_finetuning.sh. The two disagree.
• LoRA alpha. Code reads model_args.lora_alpha train.py but the argument is declared only for is_lora, lora_rank and target_modules train.py. The LoRA script also takes TARGET_MODUILES as a placeholder LoRA_finetuning.sh. The LoRA path does not run as shipped without edits.
• Chunking. Paper: 2048 tokens, 512 overlap §4.5. Code uses stride 1536 prepare_tokenized_dataset.py. Matches, but see Pitfalls for the last window.
• Filter. Thresholds in code filter.py match app. C.1. The paper's rule is strict '>' alg. 1; the code uses '>=' filter.py.
• BLEU. Paper labels rationale scores as BLEU-1 tab. 2 and writes the equal-weight BLEU-4 formula eq. 2. Code calls sentence_bleu with no weights rationale_similarity.py. Its default is equal weights over 1- to 4-grams, so the value is not BLEU-1 unless weights are passed. Tokenisation is language-specific: word_tokenize for en, es, fr and ru, jieba for zh, and a Japanese tokeniser for ja rationale_similarity.py. The paper does not state tokenisation.
• ROUGE. Paper labels rationale scores ROUGE-1 tab. 2. Code returns the ROUGE-L F-score rationale_similarity.py. Those two do not match.
• BERTScore. Code uses bert-base-multilingual-cased rationale_similarity.py, which matches §4.3.
• Instructions. Appendix says each fine-tuning sample is seen under two instructions app. A. Code has two instruction strings train.py train.py. Consistent.
• Llama 3. Paper trains MMed-Llama 3 §2; the repo has scripts for InternLM and InternLM 2 only. No Llama 3 pre-training recipe is in the code.

Pitfalls

• Llama 3 continued pre-training is not reproducible from the paper or the code. The paper's LRs cover InternLM and BLOOM only §4.5, and the repo has no Llama 3 script. Its LR, batch and steps are "not stated". Any implementer should treat the headline MMed-Llama 3 numbers as author-reported and not re-derivable.
• Last chunk overlap is not fixed at 512. The loop builds the final window as tokens[-max_len:] prepare_tokenized_dataset.py. The overlap between that window and the previous one is at least 512 and can approach 2048 for streams just past a window boundary. Account for this when computing epochs or unique tokens.
• Short documents are padded with the pad token to 2048 prepare_tokenized_dataset.py. The paper says nothing about padding in pre-training.
• Rationale metric labels. The tables say BLEU-1 and ROUGE-1; the code computes BLEU with default 4-gram weights and ROUGE-L F [see Paper vs code]. Compare numbers only with the same metric code.
• Mixed evaluation setups in English tables. Scores for models other than the authors' come from original papers or the Open Medical-LLM leaderboard tab. 3. Prompting and decoding are not identical across rows, and the paper says it uses basic generation without CoT or self-consistency tab. 3.
• Arithmetic in the ablation text. The paper says rationales add 4.06 points, from 58.72 to 62.79 §2. The table values give 4.07.
• Synthetic rationales. The reference rationales are GPT-4 output that passed a check on 200 items per language §4.3. The 5.3% that failed are not described. A low BLEU or ROUGE may reflect the reference rather than the model.
• Closed-model comparison. GPT-4 (5-shot, CoT) averages 79.43 tab. 1. The paper states that CoT and few-shot prompts cost more tokens §4.3. Closed models were run through a public API; the paper says their training data are unknown, so "zero-shot" is not verified §2.
• Cost. Auto-regressive training of the 7B model took about 20 days on 8 A100 GPUs §4.5. Corpus construction needed OCR for the books and GPT-4 calls for rationales and topics §4.1 §4.3.
• Licences of released artefacts differ from this card's licence. MMedC and MMedBench data are released under CC BY-NC-SA, which is not commercial use §6. The code is CC BY-SA §7. The InternLM-based weights are Apache-2.0, and MMed-Llama 3 is under the Llama 3 community licence §7. This card uses the paper's text, not the data.
• Scope of the benchmark. MMedBench covers six languages and five QA sources §4.3. The English benchmark is a separate test with other datasets §4.4. Per-language sizes are listed in app. D.
• Training and test data are from one benchmark. Fine-tuned open models were trained on the MMedBench training split and tested on its test split tab. 1. Those numbers measure in-benchmark fine-tuning, not performance on other medical sources.

Lineage

• Builds on: the medical book collection approach of PMC-LLaMA §4.1. No card.
• Compared against: GPT-4, GPT-3.5, Gemini 1.0 Pro, Flan-PaLM, Med-PaLM 2, BLOOM, InternLM and InternLM 2, Llama 2 and Llama 3, Mistral, MEDITRON, BioMistral, MedAlpaca, ChatDoctor, PMC-LLaMA and Gemma §4.3 §4.4.
• Superseded by: not known from wiki content.