GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

In plain words

A language model on its own cannot reliably recall exact database entries, such as which chromosome a gene sits on. These answers live in specialised databases that search engines do not index well, so a chat model that browses the web does poorly on them too.

GeneGPT takes a different route. It does not retrain the model. It writes a long prompt that explains how to ask the U.S. National Center for Biotechnology Information (NCBI) for data over its web addresses, and it includes a few worked examples of such requests. While the model writes its answer, a small controller watches for a special arrow that the model writes when it wants a lookup. The controller stops the model, runs the web address, pastes the raw reply back into the text, and lets the model carry on.

The point is that the model reads real database records instead of recalling them. On nine genomics question sets, the authors report a higher average than the other models they compare, though not on every task. The results are the authors' own.

TL;DR

GeneGPT is a prompting-plus-decoding method, not a fine-tuned model. Codex (code-davinci-002) gets API documentation and demonstrations in context, and a custom loop executes the NCBI E-utils and BLAST URLs the model emits §2.2 alg. 1. The authors report an average of 0.83 on nine GeneTuring tasks, against 0.44 for New Bing tab. 2. Four of the in-context demonstrations are questions taken from the scored test sets, so treat the one-shot scores with care (see Pitfalls).

Problem

Language models without tools cannot retrieve specific database entries such as gene symbols, SNP locations or alignment coordinates; the authors say such data-intensive questions need precise recall that auto-regressive models lack §1 §5. Retrieval-augmented search also falls short, because specialised databases are usually not indexed by commercial search engines §5. Sequence alignment is singled out as almost impossible for a language model alone §3.4. The method assumes the model can produce correctly formed API URLs from a prompt and that the NCBI web services are reachable at run time §2.1 §2.2.

Method

GeneGPT has two parts: an in-context prompt and a decoding loop that executes API calls §2.

Prompt (in-context learning) §2.2. The prompt has four parts, in this order: an instruction, API documentations, API demonstrations, and the test question. The first three are fixed for all tasks §2.2 app. A main.py.

1. Instruction: a one-sentence task description, followed by the documentation and demonstration blocks summarised in tab. 1.
2. Documentation Dc.1 describes the E-utils endpoint: its three functions (esearch, efetch, esummary), its databases (gene, snp, omim), and the usual two-step pattern of searching for identifiers and then fetching records app. A §2.1.
3. Documentation Dc.2 describes the BLAST URL API: a PUT request returns an identifier that a later GET uses to fetch results app. A §2.1.
4. Demonstrations Dm.1 to Dm.4 are worked question-answer pairs. Dm.1 is a gene-alias lookup (esearch then efetch), Dm.2 an SNP-to-gene lookup (esummary on snp), Dm.3 a disease-to-gene lookup (esearch then esummary on omim), and Dm.4 a BLAST alignment §2.2 tab. 1 app. A.
5. Each API URL and its result are wrapped in square brackets, and an arrow symbol sits between them. The arrow is the trigger for an API call §2.2.

Two prompt variants are used. GeneGPT-full uses all documentation and all four demonstrations. GeneGPT-slim uses only Dm.1 and Dm.4 §3.2 §4.1.

Decoding loop §2.3 alg. 1. The question is appended to the prompt, and Codex generates greedily. When the arrow appears, the loop extracts the last URL, calls the NCBI API, appends the raw result, and resumes generation. When an answer marker appears, the loop stops and returns the text after "Answer:" §2.3. The loop as implemented differs from this description in a few places (see Paper vs code).

# GeneGPT decoding loop  [alg. 1] [§2.3]
# Input:  question (str), header (str): instruction, Dc, Dm
# Output: answer (str)

def answer_question(question, header):
    # Prompt = header + "Question: " + question  [§2.2] [code: main.py#L139]
    prompt = header + "Question: " + question + "\n"
    while True:
        # Decode at temperature 0 until the arrow  [§2.3]
        # [code: main.py#L152-L158]
        text = codex_generate(prompt, stop=["->", "\n\nQuestion"])
        prompt = prompt + text
        if not text contains an API URL in square brackets:
            # No call requested: the text is the answer  [§2.3] [code: main.py#L199-L202]
            return text
        # Take the URL immediately before the arrow  [§2.3] [code: main.py#L181-L184]
        url = last_bracketed_url(text)
        # Run the URL on NCBI; cap the reply length  [§2.3] [code: main.py#L189-L196]
        result = call_ncbi(url)
        if result is a BLAST PUT reply:
            # Keep only the request identifier  [app. A] [code: main.py#L191-L193]
            result = request_identifier(result)
        prompt = prompt + "->[" + result + "]\n"

Pitfalls of the loop are covered under Paper vs code and Pitfalls. The paper does not state the per-question call limit, the result cap, or the context truncation, which the code applies (see Hyperparameters).

Evaluation §3.3. Predictions are compared with the gold answers using exact string matching for most tasks. Gene disease association and the disease-linked gene location task use per-gene recall. The DNA-to-human-genome task gives half credit when the chromosome is right and the position is wrong, because the benchmark names no reference build §3.3 app. D evaluate.py.

Tasks §3.1 §4.2. GeneTuring (Hou et al.) has 12 tasks of 50 question-answer pairs each. The authors use nine NCBI-related tasks, grouped as nomenclature, genomic location, functional analysis and sequence alignment §3.1 tab. 2. They also add GeneHop, a new three-task multi-hop set of 50 questions each: SNP gene function, disease gene location, and sequence gene alias §4.2 app. C.

Hyperparameters

Results

Author-reported. Setup: GeneTuring, nine tasks, 50 questions each; Codex (code-davinci-002) with the in-context prompt; baselines reported by the GeneTuring authors, who evaluated them manually §3.1 §3.2 §3.3. The GeneGPT numbers use the stricter automatic evaluation described above. Full per-task numbers are in tab. 2. Scores are exact-match accuracy except where noted.

GeneHop, multi-hop, averages over three tasks tab. 3:

Error analysis: the authors classify errors into wrong API, wrong arguments, failure to extract the answer, answer absent from the API result, and other. Simple alias and location tasks fail mostly with the last type, disease tasks show wrong-API errors, alignment tasks fail mostly as "other" §4.3 tab. 4 app. E.

Ablation: documentations alone fail on all tasks; the alias demonstration alone performs comparably to full on most tasks; the BLAST demonstration is what makes alignment work §4.1 fig. 2.

Paper vs code

Code: ncbi/GeneGPT at commit 0a2a327a507be49e660597769cc53431abd6613a (owner NCBI, the authors' institution; the README cites this work's Bioinformatics article). Code license: NCBI public domain notice (US Government work), a non-standard notice in the repository LICENSE file. The README documents python main.py 111111 for full and python main.py 001001 for slim README.md.

• Stop sequences. The paper says generation stops on "->" and on the answer marker "\n\n" §2.3. The code stops on "->" and "\n\nQuestion" main.py. There is no separate answer-marker stop. The answer is the final model text, returned when no URL was generated main.py. The answer string then goes through get_answer in evaluation evaluate.py.
• Context window. The paper gives Codex an 8k-token window §2.3. The code keeps the last 18000 characters and cuts from the front main.py main.py. The paper does not state this truncation.
• Call and reply limits. The code caps each question at 10 API calls and each reply at 10000 characters main.py main.py. Neither limit appears in the paper.
• BLAST identifier. The appendix caption for Dm.4 says a regular expression extracts the PID and appends it to the prompt app. A. The code extracts the RID with RID = (.*)\n and uses it as the reply main.py main.py. Both are request identifiers, but the names differ.
• Demonstrations are fetched live. The prompt header is built at run time by calling NCBI: the demonstration results, including the BLAST request identifier and the 30-second wait, come from the live services main.py. The prompt the paper shows is a snapshot, and a rerun can receive different text.
• Gene location scoring. The paper says exact match for the location tasks §3.3. The code takes the last whitespace-separated token of the reply and prepends "chr" if missing evaluate.py. That is more lenient.
• Disease gene association and disease gene location scoring. The paper says recall over exact individual gene matches §3.3. The code checks whether each gold gene string is a substring of the whole reply evaluate.py. Substring matching can credit a gene whose name contains the gold symbol.
• Mapped answers. Yes/No to TRUE/NA for the protein-coding task and the species-name mapping are implemented as the paper describes evaluate.py §3.3.
• Chat-model variant. main_turbo.py runs the same loop on gpt-3.5-turbo-16k at temperature 0 with a 36000-character cut and the same two stop strings main_turbo.py main_turbo.py. The paper reports only Codex results, and the README documents main.py.
• Result files. The README's printed scores for GeneGPT-slim (for example 0.84 for gene alias, 0.66 for disease association, 0.44 for human DNA alignment, 0.88 for multi-species alignment) match the GeneGPT-slim column of tab. 2. These were compared on the printed values; the card did not re-run the evaluation.
• Dependencies. The code pins openai==0.27.7, which is the legacy completion interface needed for Completion.create requirements.txt main.py. Neither the paper nor the code states whether code-davinci-002 is still served; check access before planning a rerun.
• Repository license file. The LICENSE text is the NCBI public domain notice and asks citation of a different paper (BioCPT). This is attribution text inside the repo, not a restriction; cite this paper's authors LICENSE.

The four demonstration questions and answers in the code match the paper's appendix examples main.py app. A. The GeneHop driver is not in main.py; the repository's GeneHop results are stored as files and were not re-checked.

Pitfalls

• Four demonstrations are scored test items. Each demonstration question is a GeneTuring test question, and its gold answer sits in the same data file the evaluator scores geneturing.json geneturing.json geneturing.json geneturing.json. The four are: the LMP10 gene-alias question (answer PSMB10, Dm.1), the rs1217074595 SNP question (answer LINC01270, Dm.2), the Meesmann corneal dystrophy disease question (answer KRT12, KRT3, Dm.3), and the DNA alignment sequence (answer chr15:91950805-91950932, Dm.4). The paper says the demonstrations are "based on questions in the GeneTuring tasks" §2.2 and that the same questions are the test questions app. B. The evaluator does not exclude them main.py evaluate.py. Each one is one of 50 items in its task, so it moves that task's average by 0.02 if answered correctly. Slim keeps Dm.1 and Dm.4, so gene alias and the human-genome alignment include a demonstrated item. Re-score with those four items excluded before comparing with other systems.
• Task count. The abstract and introduction say eight tasks [abstract] §1. Section 3.1 says nine tasks and Table 2 lists nine §3.1 tab. 2. Use nine.
• Which tasks are one-shot. Section 3.4 says the starred tasks are one-shot and then says GeneGPT is "all 2 one-shot tasks" §3.4. Table 2 stars four rows in the full column and two in the slim column tab. 2. Check the column before reading one-shot as a property of a task.
• Slim was picked on the test benchmark. GeneGPT-slim comes from ablation and probing on GeneTuring itself §3.2 §4.1. The paper describes no held-out split. Its 0.83 is the best overall figure in the table, so read it as an in-sample selection.
• Baselines are not evaluated the same way. The paper says its evaluation is stricter than the manual evaluation used for other models, which may accept non-exact matches §3.3. The comparison is therefore not like-for-like, and the authors state this themselves.
• Context truncation drops the header first. When the prompt exceeds the cut, the code removes the earliest characters main.py. The instruction, documentation and demonstrations come first in the prompt, so long multi-hop chains lose them before the question. Keep chains short or raise the window.
• Live services change the prompt. Demonstration replies, the BLAST request identifier and the BLAST wait are fetched at run time main.py. The paper notes that the BLAST interface was about to be deprecated and used a regular expression to parse its first step app. A. Plan for failures in those calls, and log the prompts sent main.py.
• Retry and failure paths. An overlong request is recorded as "lengthError" and scored as wrong; 10 calls per question produce "numError" main.py main.py. Count those failures before reporting a score.
• Sequence alignment depends on a reference build. The benchmark names no reference genome, so the code scores chromosome-only matches at half credit §3.3 app. E. Position mismatches (e.g. chr8 at 7081648 vs 1207812) are classed as "other" errors app. E.
• Codex access. The method requires the code-davinci-002 engine through the legacy OpenAI completion interface §2.3 requirements.txt. The paper and code do not say whether this engine is still served. A reimplementation needs a different model and must re-measure the baselines it depends on.
• Not for clinical use. The repository README states that its outputs are not intended for diagnosis or medical decisions without professional review README.md. Keep that caveat in any product built on this method.

Lineage

• Builds on: Codex (code-davinci-002) for code-aware generation §2.3, cites Chen et al. 2021; in-context learning §2.2, cites Brown et al. 2020; the GeneTuring benchmark §3.1, cites Hou et al. 2023.
• Related tool-augmentation work named by the paper: PAL, TALM and Toolformer §5.
• Compared against: GPT-2, GPT-3 (text-davinci-003), ChatGPT, BioGPT, BioMedLM and New Bing, as reported by the GeneTuring authors §3.2 tab. 2.
• Superseded by: not known from wiki content.