DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task

In plain words

Large chatbots such as ChatGPT answer medical questions in English much better than in Chinese, and they were not trained for medicine, so their medical advice can be unreliable. This paper takes an open, mid-sized model that already handles Chinese (ChatGLM-6B) and teaches it to answer patient questions using Chinese doctor-patient dialogues.

Training does not change all of the model's weights. It adds a small set of new weights on top and updates only those, which is what lets one data-centre GPU do the job. The authors also translated an English medical chat dataset into Chinese, using ChatGPT's translations to train a smaller translation model so they did not have to pay ChatGPT for every sentence.

The result is an early model that the authors openly call rough, and they ask for feedback. The main claim is that a medical assistant like this can be trained and run cheaply, so hospitals could try it on their own data. The evaluation is a handful of example answers marked up by a doctor, not a benchmark.

TL;DR

DoctorGLM fine-tunes ChatGLM-6B (6.2B parameters) on Chinese medical dialogues with LoRA, rank 8 and alpha 16 on the query and value projections, for one epoch with the Lion optimizer at learning rate 2e-5 §3.3. It also translates the English ChatDoctor data using a small model trained on ChatGPT translations §3.1. No quantitative benchmark is reported; the evidence is doctor-marked example answers tab. 2 tab. 3 tab. 4. The authors say the model "contains various mistakes" and is not for clinical use [abstract] §6, and the released repository says LoRA was later dropped for P-tuning README.md.

Problem

• General LLMs are mostly trained in English and not for medicine, so their diagnoses and drug advice can be imprecise §1.
• Translating a large English medical set with ChatGPT would cost "tens of thousands of dollars", which the authors say is unaffordable for most researchers §3.1.
• Hospitals are assumed to be unable to train or deploy a dialogue model [abstract]. The paper's claim is that fine-tuning a 6B model is feasible on one A100 [abstract] §3.3.
• Assumes a base model that already reads Chinese (ChatGLM-6B, pretrained on about 1 trillion Chinese and English tokens per the authors) §3.3.

Method

Four parts: a dataset, a translation step, an optional prompt designer, and the LoRA fine-tune. Inference sampling is a fifth, minor part.

1. Training data §2 tab. 1. The core data is Chinese medical dialogue (CMD) from five departments, all single-turn question and answer pairs. The paper does not say where the CMD data comes from; the repository links a third-party collection README.md. The department counts in tab. 1 sum to about 882K pairs (our sum; the paper does not state a total).

Column meanings: "Chat" is multi-round QA, "Syn." is data generated by other language models tab. 1.

2. Translating ChatDoctor §3.1 fig. 2. A subset of English ChatDoctor pairs is sent to the ChatGPT API to get reference Chinese translations. A pre-trained BART-style translation model is then fine-tuned on those pairs only. The fine-tuned model translates the English data, which is then used for training. The paper does not state the subset size or any training settings for the translation model.

# Distil ChatGPT translations into a small model  [§3.1] [fig. 2]
# Input:  english_texts: subset of ChatDoctor text, in English
# Output: translator: small model, English to Chinese

def train_translator(english_texts):
    # Ask ChatGPT for a reference Chinese translation of each text  [§3.1]
    pairs = []
    for text in english_texts:
        chinese = chatgpt_translate(text)
        pairs.append((text, chinese))
    # Fine-tune a pre-trained BART-style model on these pairs only  [§3.1]
    translator = fine_tune_bart(pairs)
    return translator

3. Prompt designer, optional §3.2. Before the model answers, a module extracts disease keywords from the user's question, picks the most likely disease name, and pulls a short description from a library of disease documents. The description is combined with the original question. The paper says the library has 3231 documents from the Merck Manual §3.2. The paper gives no matching algorithm.

# Prompt designer, optional  [§3.2]
# Input:  question: str, the user's message
# Output: prompt: str, the question plus a disease description

def build_prompt(question, model, disease_library):
    # Extract keywords and name the most likely disease  [§3.2]
    disease = model_name_disease(model, question)
    # Get the short description from the disease library  [§3.2]
    description = disease_library.describe(disease)
    # Combine the description with the original input  [§3.2]
    return description + question

4. LoRA fine-tuning of ChatGLM-6B §3.3. The base model is frozen. Low-rank adapters are added to the query and value projections, with rank 8 and alpha 16. Only the adapter weights are trained, giving about 7 million trainable parameters per the paper (see Pitfalls). Training uses the Lion optimizer at learning rate 2e-5, with no warmup and no weight decay, for one epoch at batch size 4. Inputs are cut to 512 tokens and targets to 100 tokens.

# LoRA fine-tuning of ChatGLM-6B  [§3.3]
# Input:  base: ChatGLM-6B, dialogues: list of (question, answer)
# Output: base with trained LoRA adapters

def finetune_lora(base, dialogues):
    # Freeze the base; add rank-8 adapters with alpha 16  [§3.3]
    # Query and value only  [§3.3]
    # The code stores q and v as slices of one fused QKV matrix
    # [code: Doctor_GLM/ckpt/lora/adapter_config.json#L3-L7]
    add_lora_adapters(base, rank=8, alpha=16, targets=["q", "v"])
    freeze_all_except(base, lora_parameters)
    # Lion optimizer, learning rate 2e-5, no weight decay, no warmup  [§3.3]
    optimizer = lion(lora_parameters, learning_rate=2e-5)
    # One epoch over the data, batch size 4  [§3.3]
    for batch in batches(dialogues, size=4):
        # Truncate input to 512 tokens and target to 100  [§3.3]
        inputs, targets = tokenize(batch, max_input=512, max_target=100)
        # Loss: cross-entropy, named only in the limitations  [§6]
        loss = cross_entropy(base(inputs), targets)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    return base

5. Sampling at inference §4.2 tab. 6. The paper tests temperature and top-p (nucleus) sampling. Temperature scales the randomness of the next-token distribution. Top-p keeps the smallest set of most likely tokens whose probability reaches a threshold, then samples from it. The paper does not state the order in which these two are applied.

# Choose the next token with temperature and top-p  [§4.2]
# Input:  logits: scores over the vocabulary
# Output: one token id

def sample_next_token(logits, temperature, top_p):
    # Higher temperature gives more random output  [§4.2]
    scaled = softmax(logits / temperature)
    # Keep the smallest set whose cumulative probability reaches top_p  [§4.2]
    kept = smallest_set_reaching(scaled, top_p)
    return draw_from(kept)

Key equations: none. The paper has no numbered equations or algorithms.

Hyperparameters

Results

Author-reported. The paper has no quantitative benchmark. Setup: CMD data from five departments tab. 1; models compared are DoctorGLM, ChatGLM-6B and ChatGPT (gpt-3.5-turbo) §4.1; hardware is one A100 §3.3.

Paper vs code

Code: https://github.com/xionghonglin/DoctorGLM at commit 77635150a6b1c9992c5954538cff719e3960ad6c. The owner matches the first author's name and the paper's project page. The repository has no LICENSE file, so its license is not stated and no code may be quoted verbatim. The training script is not in the repository; the LoRA code is a third-party project named in the README README.md.

• Main method changed. The paper's main model is LoRA §3.3. The README says v0.0.2 dropped LoRA because it "keeps losing dialogue ability", and that the released weights use P-tuning README.md. The released ckpt/ptuningv2 config sets pre_seq_len to 128 config.json.
• Batch size. Paper: 4 §3.3. Template: 1 data_utils.py.
• Epochs. Paper: 1 §3.3. Template: 20 data_utils.py. The README says v0.0.1 was one epoch README.md.
• Training data. Paper: "all" CMD data §3.3. Template trains on andriatria.json only data_utils.py.
• LoRA alpha. Paper and released adapter: 16 §3.3 adapter_config.json. Template: 32 data_utils.py.
• LoRA dropout. Not stated in paper. Released adapter 0.05; template 0.1 adapter_config.json data_utils.py.
• LoRA targets. Paper: q and v §3.3. Released adapter: fused query_key_value with enable_lora [true, false, true], which means q and v only adapter_config.json. Agrees.
• Optimiser, learning rate, warmup, weight decay, lengths. Paper and template agree: Lion, 2e-5, no warmup, no weight decay, 512 input and 100 target tokens §3.3 data_utils.py. Agrees.
• Translation. Paper: fine-tunes a BART-style model on ChatGPT pairs §3.1. Code: translation.py only runs an already-released translation model, zhaozh/medical_chat-en-zh, over GenMedGPT-5k.json and drops about 1% of outputs that fail to parse translation.py translation.py. The training of the translation model is not in the repository. The script reads GenMedGPT-5k.json, and the repository does not say how that file relates to ChatDoctor.
• Prompt designer. Paper describes it as part of the method §3.2. Code seach_engine.py names the disease with the same chat model, finds the nearest disease by sentence embedding, and returns the entry from disease_info.json only if a similarity score passes 250 seach_engine.py. The threshold's unit is not explained. Grep finds no notebook that calls this module. The paper's reported examples use no prompt designer §4.1.
• Sampling. Paper gives no values for the LoRA chat. The notebook uses top-p 0.7 and temperature 0.99 chat_lora.ipynb. In the paper's own grid, (0.99, 0.7) is graded B, not A tab. 6.
• Parameter count. Paper: about 7M trainable §3.3. The released config gives hidden size 4096 and 28 layers config.json config.json. With rank 8 on q and v, that is 2 × 8 × (4096 + 4096) × 28 = 3,670,016, about 3.7M. Covering the whole fused QKV gives the same total. We did not inspect adapter_model.bin.
• Extra data. The repository ships GenMedGPT-5k.json and GenMedGLM-8k.json GenMedGPT-5k.json. The paper does not describe these, so they are not analyzed here.

Pitfalls

• Not for clinical use. The paper says the work "is unsuitable for any commercial or clinical use" §6. Do not present its outputs as medical advice.
• Training time conflicts. The abstract says 13 hours on one A100 80G [abstract]; §3.3 says 8 hours §3.3; §5 implies 3.75 hours for 100K QA pairs over 3 epochs at 80K pairs per hour §5. Our own estimate from the same throughput is about 11 hours for one epoch over the roughly 882K pairs in tab. 1. Treat every figure as author-reported and unverified.
• Trainable parameters. The 7M figure does not follow from the released LoRA settings (about 3.7M by our arithmetic above). Either the count or the target modules differ from what is written. Verify before budgeting memory or storage.
• Unit of "iter". Tab. 7 counts "iter" without defining it. With batch size 4 (paper) versus 1 (template), the same number means different amounts of data.
• Training stability. The authors report output collapse after 50K iterations §4.3 and repetition in generated text §6. The paper gives no stopping rule and says cross-entropy is "an overly rigid constraint" §6. The README also reports repetition in 100 sampled outputs README.md.
• Quantization and memory claims disagree. §1 says INT4 inference runs on an RTX 3060 (12G). §5 says inference needs about 13 GB and cites an RTX 3090. §6 says the LoRA adapter does not quantize well and INT-4 of the base uses about 6G. The README gives INT4 about 6G and INT8 about 8G, and says quantized results are still poor README.md README.md.
• Sampling order is unstated. Temperature and top-p can be applied in either order, and the paper does not say which. Check the library's defaults before reproducing tab. 6.
• Evaluation is thin. Tab. 6 is one question and one doctor's grades; the number of doctors and samples is not stated. Tabs. 2 to 5 are four questions, with no counts.
• Unfinished sentences in the text. §4.1 ends mid-sentence ("ask a doctor to mark the error and"), and the Q2 comparison trails off ("ChatGLM suggested to use Ribavirin Granules,").
• Training data provenance. The CMD source is named only in the README README.md, and its licence is not checked here. Do not republish the data from this card.
• Code licence missing. No LICENSE file exists in the repository, so only paraphrased pseudocode is used on this card.
• Figure and table numbers are derived from LaTeX source order, not from the compiled PDF. If in doubt, cite the section.

Lineage

• Builds on: ChatGLM-6B, a bilingual GLM base model cited as du2022glm and zeng2023glm-130b in the paper §3.3. Neither has a card here.
• Builds on: ChatDoctor, whose English medical chat data the paper translates, cited as arXiv:2303.14070 (card) §3.1.
• Tested alongside: P-tuning v2, arXiv:2110.07602, which the authors test as the alternative to LoRA §4.4. No card.
• Compared against: ChatGPT (gpt-3.5-turbo) and ChatGLM-6B §4.1.
• Superseded by: not stated in this wiki's content.