HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge

In plain words

General chatbots are trained mostly on English text. They give confident but unreliable answers to medical questions, and a wrong drug suggestion or diagnosis can cause harm. HuaTuo is a language model built on LLaMA-7B, a base model, and then trained further on Chinese medical questions and answers.

The questions and answers were not written by hand. The authors took facts from a Chinese medical knowledge graph, which stores entries about diseases, drugs and symptoms. They sampled entries and asked a large language model, through the OpenAI API, to write a question and answer based on each one. The base model was then trained on those pairs. It is like writing practice questions from a reference book and then training a student on them.

The authors had people with medical backgrounds rate answers from several models. The ratings covered whether an answer could mislead the user, how much medical expertise it showed, and how smooth the language was. HuaTuo scored best on usability and smoothness in that test and stayed close to the base model on safety. These are the authors' own ratings on their own questions, so read them as a first signal, not as proof.

TL;DR

HuaTuo supervised-fine-tunes LLaMA-7B on over 8,000 Chinese medical question-answer pairs generated by the OpenAI API from sampled entries of the CMeKG knowledge graph §3.2 §3.3. In the authors' three-point annotator study, HuaTuo has the best usability and smoothness of the four models and is close to LLaMA on safety tab. 4. The paper states almost no training hyperparameters; the official repository supplies them, but it has changed since the paper (see Paper vs code).

Problem

General-domain LLMs often lack the expert knowledge that medical answers need. The paper says this can lead to poor diagnostic precision, drug recommendations and medical advice §1. Earlier work mostly supplied medical information retrieved from conversations, where human errors may be more frequent §1. Most LLMs are trained mainly in English, which weakens their Chinese responses §1. The paper also argues that the factual correctness of responses matters more in biomedicine than the diversity of generated instructions §3.3.

Method

The pipeline has three parts: a base model §3.1, a knowledge source §3.2, and instruction data built from that knowledge, used for supervised fine-tuning §3.3.

Base model. LLaMA-7B, chosen for its accessibility to train §3.1.

Knowledge source. The authors use CMeKG, a Chinese medical knowledge graph that gives structured knowledge about diseases, drugs and symptoms §3.2. Each entry is a set of named fields. tab. 1 shows one example per type: a disease entry lists drug treatments, high-risk factors, affected sites, auxiliary examinations and medical history; a drug entry lists its class, indications and ingredients; a symptom entry lists related diseases and symptoms, examinations, departments and affected sites. tab. 1 gives each example in Chinese and in English translation. The introduction says the model integrates both structured and unstructured medical knowledge §1; Method describes only the graph §3.2.

Instruction data. The steps, as the paper describes them §3.3:

1. Sample knowledge instances from the graph. The sampling rule is not stated.
2. Prompt the OpenAI API to write a question and an answer grounded in each sampled instance. The prompt wording, the model name and the number of prompt variants are not stated.
3. Keep the question as the input and drop the instruction. The paper contrasts this with generic instruction tuning, where each example has an instruction such as "Answer the following question" §3.3 tab. 2. tab. 2 shows a generic example (a translation task) for illustration only.
4. Collect over 8,000 instances for supervised fine-tuning §3.3. tab. 3 shows one example: a migraine case, with a question about treatment and an answer that covers drug and non-drug therapy.

# Build question-answer pairs from sampled knowledge entries  [§3.2] [§3.3]
# Input:  entries: sampled CMeKG records (disease, drug or symptom)
# Output: pairs of (question, answer), over 8,000 in total  [§3.3]

def build_pairs(entries):
    pairs = []
    for entry in entries:
        # Ask the OpenAI API for a question and an answer grounded in
        # this entry; the prompt wording is not stated  [§3.3]
        question, answer = generate_qa(entry)
        pairs.append((question, answer))
    return pairs

# Supervised fine-tuning of LLaMA-7B on the pairs  [§3.1] [§3.3]
# Input:  base_model (LLaMA-7B), pairs
# Output: tuned model; optimizer, learning rate and loss are not stated

def fine_tune(base_model, pairs):
    model = base_model
    for batch in batches(pairs):  # batch size not stated
        # Keep only the question as input; no instruction text  [§3.3]
        prompts = [question for question, answer in batch]
        answers = [answer for question, answer in batch]
        # Train the model to produce each answer after its prompt  [§3.3]
        loss = language_model_loss(model, prompts, answers)
        model = update(model, loss)  # update rule not stated
    return model

Hyperparameters

The paper states the base model and the data size only. The rows below marked "code" come from the official repository at code_commit. They are the script and default values, not values the paper reports. The Where column gives the source.

Results

Author-reported. Setup: the authors wrote a test set of potential questions in Chinese dialogue scenarios; five annotators with medical backgrounds rated randomly mixed responses from four models on Safety, Usability and Smoothness, each on a scale from 1 (not acceptable) to 3 (good), with 2 meaning acceptable §4.3. Baselines are LLaMA (7B), Alpaca and ChatGLM-6B §4.1. Values are average scores tab. 4.

Bold marks the best value in each column, as in the paper tab. 4. The text says HuaTuo "significantly" improves usability without much loss of safety §4.3. No significance test, variance, number of questions or annotator agreement is reported.

Paper vs code

Code at code_commit (ba9b3d65b33ba346e3d60979019abe6fa2ac741a) is the repository's current main branch, which has changed since the paper's April 2023 arXiv version. Differences:

• Instruction text. The paper drops the instruction and keeps only the input §3.3. The prompt template adds a fixed Chinese sentence before each question, roughly "Below is a question; use medical knowledge to answer it correctly", and ### 问题: and ### 回答: markers med_template.json. The data file puts the question in instruction and leaves input empty llama_data.json.
• Tuning method. The paper does not name one. Code and README use LoRA finetune.py README.md.
• Training hyperparameters. The paper gives none. The script values are in the Hyperparameters table finetune.sh.
• Base checkpoint. The paper says LLaMA-7B §3.1. The script loads decapoda-research/llama-7b-hf finetune.sh. The paper does not say which checkpoint it used.
• Data generation. The paper says the OpenAI API §3.3. The README names GPT-3.5 README.md and says the knowledge-base and data construction code is still being organised README.md. The generation prompts are not in the repository.
• Knowledge sources. The paper uses CMeKG only §3.2. The README describes public and self-built knowledge bases README.md, and later medical-literature data in data-literature/liver_cancer.json README.md. Neither is part of the paper's method.
• Evaluation. The paper's SUS test set is not in the repository. The benchmark/ folder holds CMCOQA, a later benchmark of 1,200 questions scored by people and GPT-4o README.md. It is not the paper's evaluation.
• Decoding. The paper states no decoding settings for evaluation. The inference defaults are in infer.py; the Gradio demo uses 128 new tokens generate.py.
• Name. The paper calls the model HuaTuo. The README records a rename to BenTsao (本草) on 2023-05-12 README.md, and the demo title is BenTsao generate.py.
• License. LICENSE is the Apache-2.0 text, but its copyright line names other authors LICENSE. This card records code_license as Apache-2.0 from the file.

Pitfalls

• Training recipe is missing from the paper. Method, optimizer, learning rate, epochs, batch size, LoRA or full tuning, and loss are not stated §3.3. The repository values can be used to reproduce, but they are a later state of the code and are not verified against the paper's run.
• Question generation is not reproducible from the paper. The sampling rule, the prompt and the model name are not stated §3.3. The repo does not include the generator.
• The data has known errors. The README says the training set still contains errors and imperfections README.md. The Ethics Statement says that accuracy of model responses cannot be guaranteed (Ethics Statement).
• Evaluation is small and unpublished. Test set size, sampling, annotator agreement and statistical tests are not reported §4.3. Scores are averages on a three-point ordinal scale, so differences of 0.1 to 0.2 need care.
• "Significantly" has no test behind it. The paper uses the word §4.3 but reports no test.
• Safety is not an improvement. LLaMA scores highest on safety (2.93) and HuaTuo is at 2.88 tab. 4. Usability and smoothness are the gains.
• Scope. Chinese only; 7B model only; one knowledge graph; medical advice is explicitly not the intended use (Ethics Statement).
• Terms for reuse. The paper does not state the terms that apply to the OpenAI-generated training data or to the LLaMA base weights. Check both before reusing the data or a model trained on it.
• Table 2 is not HuaTuo data. Its translation example illustrates generic instruction tuning tab. 2. Do not read it as a HuaTuo training instance.
• Dependencies. The repo pins transformers 4.30.1, peft 0.3.0 and bitsandbytes 0.37.2 requirements.txt. Newer versions may change LoRA or 8-bit behaviour; this is not tested here.

Lineage

• Builds on: LLaMA as the base model §3.1 (bibliography entry touvron2023llama, arXiv:2302.13971) [/papers/2302.13971]. Instruction data generated from examples follows the approach of Self-Instruct (cited as wang2022self, not carded here) §3.3.
• Compared against: LLaMA-7B, Alpaca and ChatGLM-6B §4.1. Alpaca's version and ChatGLM's training data are not stated in the paper.
• Related later work, not carded here: the README cites a knowledge-tuning paper (arXiv:2309.04175) and a medical-literature paper (arXiv:2309.04198) README.md README.md.
• Superseded by: not stated in the paper.