ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge

In plain words

General chatbots were not trained specifically on medical dialogue, and their medical answers can be incorrect. The authors took an open model (LLaMA) and trained it further on a large set of real conversations where patients described their symptoms and human doctors replied. That teaches the model the style and concerns of medical advice.

They also gave the model a notebook to consult. For each question, the model first lists search words, then looks up matching passages from Wikipedia or from a table of diseases, reads the most relevant ones, and writes its answer from them. The point is to make answers more specific and to cover newer terms the training data may lack. The authors stress that this is a research prototype, not a clinical tool.

TL;DR

• Fine-tunes LLaMA-7B on about 100k HealthCareMagic patient-doctor dialogues p. 2 p. 5.
• Adds an autonomous retrieval loop: keyword extraction, term-count ranking, and reading of top sections, over Wikipedia or a disease database p. 3 p. 4 p. 5.
• On 10k iCliniq questions, BERTScore F1 is 0.8446 (ChatDoctor) vs 0.8406 (ChatGPT); the margin is small and the paper reports it as author-reported tab. 1 p. 7.
• The paper and the released code disagree on several hyperparameters and on the number of retrieved sections. Check the Paper vs code section before reimplementing.

Problem

Large general-purpose chat models were not trained specifically on medical dialogue, and their medical answers can be incorrect p. 2. Hallucinated medical content is treated as a serious risk p. 11. Models frozen at training time also miss recent medical terms; the paper's example is Mpox, which the baseline could not recognise p. 5 fig. 7. The authors propose domain dialogue fine-tuning plus an external knowledge source to address both limitations p. 2 p. 3.

Method

Stage 1: dataset. Collect about 100k patient-doctor conversations from an online consultation site (HealthCareMagic-100k). Remove conversations that are too short, manually drop replies with errors, strip identifying information and correct grammar with an automatic tool p. 2. The filter thresholds are not stated. A separate set of about 10k iCliniq conversations is kept for testing, sampled across medical specialties p. 2.

Stage 2: base model and fine-tuning. Start from LLaMA-7B, a decoder-only transformer p. 5. First fine-tune on Stanford Alpaca's 52K instruction data for basic conversation skills, then refine on HealthCareMagic-100k p. 2 p. 5. The training recipe is in Hyperparameters.

Stage 3: knowledge source. Two sources are used. (a) Live Wikipedia pages. (b) An offline disease database, built from MedlinePlus, whose entries list a disease, its symptoms, further tests and treatments p. 3 p. 4 fig. 2 fig. 3. The paper notes Wikipedia is not fully reliable and that more authoritative sources could replace it p. 3.

Stage 4: autonomous retrieval at inference. The model is driven by three prompts fig. 4 fig. 5 fig. 6. The steps below follow the paper; the pseudocode shows the same flow.

1. Keyword prompt: ask the model for comma-separated search keywords for the patient's question fig. 4 p. 4.
2. Ranking: retrieve sections that contain the keywords and rank them by the number of keyword hits. The paper's retrieval is a term-matching system p. 4 p. 5.
3. Reading: the model reads the top N sections one at a time and extracts what helps with the question fig. 5 p. 4. The paper uses N = 5 p. 4.
4. Final answer: give the model the question and all extracted notes, and ask for an answer fig. 6 p. 5.

# Autonomous knowledge-brain answering  [p. 4-5] [fig. 4-6]
# Input:  question (str), knowledge_brain (list of sections), model
# Output: answer (str); top_n = 5 in the paper [p. 4]

def answer_with_knowledge_brain(question, knowledge_brain, model, top_n=5):
    # 1. Ask the model for search keywords  [fig. 4]
    keywords_text = model.generate(keyword_prompt(question))
    keywords = keywords_text.split(", ")

    # 2. Count keyword hits per section and rank sections  [p. 4-5]
    hit_counts = [
        count_keyword_hits(section, keywords)
        for section in knowledge_brain
    ]
    ranked_sections = sort_by_hits_descending(knowledge_brain, hit_counts)

    # 3. Read the top N sections, keep what helps  [p. 4] [fig. 5]
    notes = []
    for section in ranked_sections[:top_n]:
        notes.append(model.generate(selection_prompt(question, section)))

    # 4. Answer from all notes at once  [p. 5] [fig. 6]
    return model.generate(final_prompt(question, notes))

Training pseudocode (the fine-tuning recipe, from the paper and code):

# Two-stage supervised fine-tuning of LLaMA-7B  [p. 5]
# Input:  base model, Alpaca pairs, patient-doctor pairs
# Output: ChatDoctor weights

def fine_tune_chatdoctor(base_model, alpaca_pairs, dialogue_pairs):
    # 1. Learn basic instruction following from Alpaca data  [p. 5]
    model = train_causal_lm(base_model, alpaca_pairs)

    # 2. Refine on patient-doctor dialogues  [p. 5]
    #    Batch 192, peak LR 2e-5, 3 epochs, max length 512  [p. 5]
    #    Warmup ratio 0.03, weight decay 0  [p. 5]
    model = train_causal_lm(model, dialogue_pairs)
    return model

Key formulas: none. The paper has no equations.

Hyperparameters

Values marked "paper" are from the paper. Values in the code are cited separately. Where the paper and the code disagree, both values are listed here, and the discrepancy is explained in Paper vs code.

Results

Author-reported. Setup: 10k iCliniq conversations as test questions, the human physician reply as reference, BERTScore as the metric, paired t-test for significance, ChatGPT as baseline, ChatDoctor (LLaMA-7B fine-tuned) as the model under test p. 7 tab. 1.

P-values per metric are in tab. 1. The paper's exact exponents are not transcribed here; check the table in the source before quoting them.

Qualitative examples (not counted, selected by the authors): knowledge-brain answers for Mpox, otitis and Daybue, where the baseline failed or was generic fig. 7 fig. 8 fig. 9 p. 5-7; four iCliniq dialogues compared with the physician reply and ChatGPT fig. 10 fig. 11 fig. 12 fig. 13 p. 8-11.

Paper vs code

Repository: https://github.com/Kent0n-Li/ChatDoctor at e77bd7ebafbffb6d82be3410cd0d4ebebe75a516, linked from the paper p. 2. The repository LICENSE is Apache-2.0 LICENSE. Its README says the model is for academic research only and that commercial and clinical use is prohibited, because it inherits LLaMA's non-commercial license README.md.

• Learning rate: paper 2e-5 p. 5; README 2e-6 README.md. The released checkpoint's log peaks at 2e-5 trainer_state.json, which agrees with the paper.
• Epochs: paper 3 p. 5; README 1 README.md. The checkpoint's log records 3 trainer_state.json.
• Batch: paper 192 p. 5; README's full fine-tune command gives 128 for 4 GPUs README.md. The LoRA command uses batch 32 README.md.
• Retrieval depth: paper reads N = 5 sections p. 4; both retrieval scripts read the top 4 csv_reader.py wiki_reader.py.
• Final-answer prompt: the paper's Figure 6 asks for an answer based on the provided information fig. 6. The code instead uses "We have provided existing answers ... The best one answer:" csv_reader.py wiki_reader.py.
• Selection prompt: the paper's Figure 5 asks the model to select relevant information fig. 5. The code uses different wording per source: a "based on the diseases ... in the Table above" prompt for the disease database csv_reader.py and a "given the context information and not prior knowledge" prompt for Wikipedia wiki_reader.py.
• Keyword prompt: the paper's Figure 4 and the code's prompt match in meaning fig. 4 csv_reader.py.
• Model size: paper LLaMA-7B p. 5; the released config matches 7B dimensions config.json.
• Chat format: the code's chat script uses a "Patient:" / "ChatDoctor:" dialogue transcript chat.py. The training script uses an Alpaca instruction template train.py. The paper does not describe either.
• Model weights: pretrained/ contains config, tokenizer and a shard index but no weight files pytorch_model.bin.index.json. The README links checkpoints on Google Drive, which were not checked.
• Dataset size: the paper's figure is about 100k; README gives the disease database as "over 700 diseases" README.md. The paper does not state the count.
• ChatGPT baseline: the paper does not give the ChatGPT model version. The repository's ChatGPT variant calls gpt-3.5-turbo chat_openai.py; this is not evidence about which model the paper used.

Pitfalls

• Conflicting training recipe. The paper and the README disagree on learning rate (2e-5 vs 2e-6), epochs (3 vs 1) and batch (192 vs 128). The released checkpoint matches the paper. Use the paper's values and treat the README as unverified.
• Checkpoint size does not match the stated data. At batch 192 the 100k dataset gives about 521 steps per epoch, or about 1,560 for 3 epochs. The released log has 609 steps. The dataset and batch used for that checkpoint are not stated.
• Keyword matching is substring matching. keyword in chunk is a plain substring test, so short keywords match inside longer words csv_reader.py wiki_reader.py. An empty keyword matches every section; model output ending in ", " produces one csv_reader.py. Strip and filter keywords before matching.
• Wikipedia lookup is by exact title. Each keyword is used as a Wikipedia page title with auto_suggest=False and the lookup errors are swallowed wiki_reader.py. Misspelled or generic keywords silently return nothing.
• Ties are not defined. Sections with equal hit counts are sorted by their text csv_reader.py. The paper gives no tie rule.
• Sampling makes output non-deterministic. Decoding uses sampling with temperature 0.5 chat.py. Fix a seed if you need reproducible evaluation; the paper does not say whether one was used.
• Section size is not in the paper. The disease database uses 6 records per section and Wikipedia uses 250 words csv_reader.py wiki_reader.py. Changing these changes which section wins.
• Loss masking differs between scripts. The full fine-tune masks prompt tokens train.py. The LoRA script trains on prompts by default train_lora.py.
• Evaluation is thin. Table 1 gives one metric (BERTScore), with physician replies as reference. The paper does not say which BERT model was used, how many questions were scored, or whether "+/-" is a standard deviation or a standard error. It also does not say whether the scored ChatDoctor used retrieval.
• Qualitative examples are selected. Figures 7 to 13 are hand-picked and show a few cases where the model did well or badly p. 5-11.
• Licensing. The paper is CC BY 4.0. The code is Apache-2.0, but the README prohibits commercial and clinical use because the model inherits LLaMA's non-commercial license README.md. Do not describe the released model as usable in products. The training data (HealthCareMagic-100k, iCliniq, Alpaca) is not in the repository, and the paper and code state no license for it.
• Not a clinical tool. The authors say the work is preliminary and that outputs can be wrong p. 11 README.md. Expect hallucinated or incorrect advice without human checks.
• Version. The fetched PDF is the Cureus journal version (15(6):e40895). The arXiv record is v5. This card reflects the Cureus PDF; the arXiv text may differ.

Lineage

• Builds on: LLaMA, arXiv:2302.13971 (base model) p. 5 p. 12. Stanford Alpaca instruction data, cited as a GitHub project p. 2 p. 12.
• Uses for evaluation: BERTScore, arXiv:1904.09675 p. 7 p. 12.
• Compared against: ChatGPT, described as a prevalent model that is not open source p. 2 p. 7.
• Retrieval: the paper cites a term-matching retrieval system for its keyword lookup p. 4 p. 12, ref. 15; the paper does not compare against other retrieval methods.