Matching Patients to Clinical Trials with Large Language Models

In plain words

Running a clinical trial needs people who fit its rules, and finding them means reading each trial's eligibility list against each patient's notes. Done by hand, that is slow and error-prone. This paper asks whether a large language model can do the job from plain text, without training on labelled examples.

The system works in three stages. First, the model reads the patient's notes and writes a ranked list of medical search terms. Those terms pull a short list of candidate trials out of a large collection. Second, for each candidate, the model goes through the trial's rules one by one. For every rule it gives a short reason, points to the sentences in the notes that matter, and says whether the patient meets it, breaks it, or whether the notes do not say. Third, those per-rule answers are rolled up into one score per trial, which is used to sort the trials and to drop the ones a patient clearly cannot join.

The point is that a person can check each step. A recruiter sees why a trial was flagged and which sentence supported it. In a small pilot, screening with the tool took less time than screening without it.

TL;DR

TrialGPT is a zero-shot patient-to-trial matching pipeline built on GPT-4 (and GPT-3.5 for comparison). It has three modules: LLM keyword generation with hybrid BM25 and MedCPT retrieval; criterion-level eligibility labels with explanations and evidence sentences; and trial-level scores from linear and LLM aggregation. Results are author-reported on three cohorts of 183 synthetic patients, with a manual check of 1,015 patient-criterion pairs from the SIGIR cohort. Relevant if you build or evaluate LLM-based screening over trial eligibility text.

Problem

Matching patients to trials is labour-intensive, slow and prone to human error p. 3. Earlier neural approaches embed patient records and criteria into a shared space for similarity search, but they need large sets of paired patient-criterion annotations, which are often unavailable, and their retrieval is hard to explain p. 4. The paper targets the patient-centric "patient-to-trial" direction: one patient, many candidate trials p. 3. Its stated gap is an end-to-end pipeline (retrieval, matching and ranking) that works without training data and gives explanations p. 4.

The evaluation assumes a pooled labelled corpus. Retrieval runs over the judged trials pooled for each cohort. The paper treats about 23,000 trials (the maximum number of active trials in one country) as the realistic search space, and notes that pooled judgments mean some eligible trials outside the labelled set are counted as irrelevant if retrieved p. 9.

Method

Input is a free-text patient note. For matching, each trial is described by its background (title, conditions, interventions, brief summary) and its inclusion and exclusion criteria p. 32. The pipeline has three modules; all use an LLM with temperature 0 p. 31.

Module 1: TrialGPT-Retrieval p. 31-32.

1. Ask the LLM for up to 32 key conditions, ranked by importance p. 31. The conditions are the keywords w_1 \dots w_K. The repo's prompt also asks for a summary of the patient's main problems, which the retrieval step ignores keyword_generation.py.
2. For each keyword w_i, rank every trial with BM25 (lexical) and with MedCPT (dense, semantic). Ranks are 1-based in the paper p. 31-32.
3. Fuse the two rankings for the same keyword with reciprocal rank fusion, and weight keyword i by 1/i so earlier keywords count more p. 32, eq. 1.
4. Sum over keywords and retrievers, and keep the highest-scoring trials as candidates p. 32.

s_j = \sum_{\mathrm{Ret}} \sum_{i=1}^{K} \frac{1}{i \times (\mathrm{Rank}(\mathrm{Ret}, w_i, t_j) + C)} eq. 1

Here \mathrm{Ret} \in \{\mathrm{BM25}, \mathrm{MedCPT}\}, C = 20, and \mathrm{Rank} = 1 means the top-ranked trial p. 32.

Module 2: TrialGPT-Matching p. 32-33. Each patient-trial pair gets two LLM calls, one for the inclusion criteria and one for the exclusion criteria p. 33. Patient sentences are numbered, and each criterion receives three outputs:

1. a relevance explanation (free-text reasoning, written first, as chain-of-thought) p. 33;
2. a list of relevant sentence IDs (empty if none) p. 33;
3. an eligibility label from a fixed set eq. 2-3.

Inclusion labels: included, not included, not enough information, not applicable. Exclusion labels: excluded, not excluded, not enough information, not applicable. eq. 2, eq. 3. Different label sets are used because exclusion phrasing is often ambiguous, and entailment labels do not separate it well p. 33.

E \in \{\text{included}, \text{not included}, \text{not enough information}, \text{not applicable}\} eq. 2

E \in \{\text{excluded}, \text{not excluded}, \text{not enough information}, \text{not applicable}\} eq. 3

Module 3: TrialGPT-Ranking p. 33-35. Two kinds of trial-level score are computed from the criterion labels.

Linear aggregation. Percentages of criteria per label, with "not applicable" criteria removed from the denominator p. 34, eq. 4-11. Example for inclusion criteria:

\%\,\text{met inclusion} = \frac{|\{E(i_x) = \text{included}\}|}{M'} eq. 4

M' = M - |\{E(i_x) = \text{not applicable}\}| eq. 7

Exclusion criteria use N' = N - |\{E(e_y) = \text{not applicable}\}| eq. 11. Equations 5-6 and 8-10 give the unmet, not-enough-information and not-excluded shares in the same way eq. 5-6, eq. 8-10.

LLM aggregation. A second LLM call reads the patient note, the trial and the criterion predictions, and outputs two scores: general relevance R and eligibility S p. 35. The paper restricts them as follows:

0 \le R \le 100 eq. 12

-R \le S \le R eq. 13

Feature combination. The ranking score adds the linear terms and the LLM scores p. 35:

\text{combination} = \%\text{met inclusion} - \mathbb{I}(\%\text{unmet inclusion} > 0) - \mathbb{I}(\%\text{met exclusion} > 0) + \%\text{LLM relevance} + \%\text{LLM eligibility} eq. 14

\mathbb{I}(\text{condition}) = \begin{cases} 1 & \text{if condition is True} \\ 0 & \text{if condition is False} \end{cases} eq. 15

Table 2's caption says a Sign() function assigns the sign per task, so excluding uses the same features with signs flipped p. 22. The excluding combination of eq. 25-26 is written for the baselines only p. 38. See Pitfalls.

Pseudocode. The following is a reading of the method in our own words. Trial and criterion IDs and the LLM helper are placeholders.

# Step 1: generate up to 32 ranked keyword conditions  [p. 31]
def generate_keywords(patient_note):
    # Ordered by priority; the paper says at most 32  [p. 31]
    conditions = llm(keyword_prompt(patient_note), temperature=0)
    return conditions[:32]

# Step 2: rank candidate trials for each keyword and fuse  [p. 31-32] [eq. 1]
def retrieve_candidates(conditions, trials, C=20):
    # Accumulates the fused score for each trial  [eq. 1]
    scores = {}
    for i, keyword in enumerate(conditions, start=1):
        # Two retrievers per keyword: lexical and dense  [p. 31]
        bm25_ranking = bm25_rank(keyword, trials)
        medcpt_ranking = medcpt_rank(keyword, trials)
        for ranking in [bm25_ranking, medcpt_ranking]:
            # Ranks start at 1 for the top trial  [p. 32]
            for rank, trial in enumerate(ranking, start=1):
                # Decay 1/i per keyword, reciprocal rank term  [eq. 1]
                contribution = 1 / (i * (rank + C))
                scores[trial] = scores.get(trial, 0.0) + contribution
    # Highest fused score first; the top ones become candidates  [p. 32]
    return sorted(scores, key=scores.get, reverse=True)

# Step 3: criterion-level matching, two calls per pair  [p. 33] [eq. 2-3]
def match_pair(patient_note, trial):
    # Sentence IDs are part of the prompt input  [p. 33]
    results = {}
    for kind in ["inclusion", "exclusion"]:
        # Each criterion: reasoning, evidence IDs, label  [p. 33]
        results[kind] = llm(match_prompt(patient_note, trial, kind))
    return results

# Step 4: trial-level scores  [p. 34-35] [eq. 4-15]
def trial_score(criterion_labels, llm_relevance, llm_eligibility):
    # Linear terms; "not applicable" leaves the denominator  [eq. 7, eq. 11]
    m_prime = count_not_applicable_removed(criterion_labels.inclusion)
    met_inc = count(criterion_labels.inclusion, "included") / m_prime
    unmet_inc = count(criterion_labels.inclusion, "not included") / m_prime
    met_exc = count(criterion_labels.exclusion, "excluded")
    # Indicator terms  [eq. 15]
    unmet_term = 1 if unmet_inc > 0 else 0
    met_exc_term = 1 if met_exc > 0 else 0
    # LLM terms are the R and S from the aggregation prompt  [eq. 12-14]
    return (met_inc - unmet_term - met_exc_term
            + llm_relevance + llm_eligibility)

Notes on the pseudocode. The LLM terms enter eq. 14 as written; the repo divides both by 100 (see Paper vs code). The pseudocode uses a raw exclusion count in the indicator. Eq. 8 divides that count by N', which does not change whether it is above zero, so the indicator value is the same either way.

Hyperparameters

Results

Author-reported. Setup: three cohorts (SIGIR 2016, TREC 2021 CT, TREC 2022 CT), 183 synthetic patients in total, GPT-4 and GPT-3.5 backbones, baselines are biomedical NLI-trained encoders p. 8-9 p. 36-37.

Cohorts tab. 1, p. 9

Retrieval p. 11-12 fig. 2

Figure 2 plots recall against depth for three retrievers and several query types (raw note, GPT-3.5, GPT-4, clinician keywords on SIGIR). Hybrid retrieval gives the best curves, and MedCPT is stronger than BM25 p. 11 fig. 2.

Matching p. 12-16 fig. 3

The manual evaluation covers 1,015 patient-criterion pairs, drawn from 105 patient-trial pairs of 53 SIGIR patients p. 12. The paper's abstract reports 87.3% accuracy on this set.

Ranking and excluding tab. 2, p. 21-22

The paper reports the GPT-4 feature combination as 43.8% above the best baseline average (0.7314 vs 0.5085) p. 23. The text gives P@10 as 0.6699 for that row p. 21; the table gives 0.6688 tab. 2. See Pitfalls.

Pilot user study fig. 5, p. 23-25

Screening accuracy was 97.2% with TrialGPT and 91.7% without, which the paper reports as not statistically significant p. 23. Time savings per trial ranged from 32.4% to 60.2%, and per patient case from 35.9% to 51.9% p. 24 fig. 5b-c.

Paper vs code

Code: official repository ncbi-nlp/TrialGPT, commit 1b3242cd50c153a9c324661d4a0bd94fc68a4226, linked from the paper's code availability statement p. 41. The repo has keyword generation, retrieval, matching and ranking scripts. It has no evaluation code (recall, NDCG, P@10, AUROC, the excluding score, the Sign convention), and results/ is empty.

• Rank convention in retrieval. Paper: Rank = 1 for the top trial p. 32 eq. 1. Code: enumerate is 0-based, so the term is rank + k with rank starting at 0 hybrid_fusion_retrieval.py. For k=20 the top hit gets 1/20 in code and 1/21 under the paper's convention.
• Retriever weights. Paper: plain fusion of BM25 and MedCPT, no weights stated p. 32. Code: the two "weights" are on/off flags (> 0), not multipliers hybrid_fusion_retrieval.py. README says to keep both at 1 outside ablations README.md.
• BM25 document field weights. Not stated in the paper. Code weights title x3, conditions x2, text x1 hybrid_fusion_retrieval.py.
• Patient note sentence IDs. Paper: sentences numbered, no mention of an extra item p. 33. Code appends a consent sentence as an extra numbered sentence to every patient note in matching and aggregation, so one evidence ID always refers to that boilerplate run_matching.py run_aggregation.py.
• Criterion parsing. Code splits criteria on blank lines, drops headers containing "inclusion criteria" or "exclusion criteria", drops fragments under 5 characters, and numbers the rest from 0 TrialGPT.py. The paper does not describe criterion numbering.
• LLM aggregation scale. Paper: eq. 14 uses "% LLM relevance" and "% LLM eligibility" p. 35. Code: both are divided by 100 rank_results.py.
• LLM aggregation restrictions. Paper states 0 \le R \le 100 and -R \le S \le R p. 35. Code: the prompt asks for these ranges TrialGPT.py, but get_agg_score only casts to float and divides, with no clamping or range check rank_results.py.
• Matching score. Code: count-based fraction for included divided by included + not included + not enough information, minus one if any not included, minus one if any excluded rank_results.py. This matches eq. 4 and eq. 7 for the inclusion side, and eq. 15 as indicators. Exclusion uses raw counts, not a fraction, so eq. 8 and eq. 11 are not what the code computes.
• Missing trials. A trial absent from the aggregation output gets score 0 for its LLM term and is still ranked rank_results.py. The paper does not state this.
• Silent failures. Matching failures are caught and skipped; the trial is then missing from the results run_matching.py. Aggregation swallows errors and skips the trial run_aggregation.py.
• Models. Paper: GPT-4 and GPT-3.5, version 0613 via Azure p. 31. Code takes the model name as an argument; README examples use gpt-4-turbo README.md README.md. The paper's 0613 snapshot is not named in the repo.
• Excluding score. Paper: eq. 25-26 define the excluding combination for baselines, and Table 2 applies a per-task sign tab. 2, p. 22. Code: only the ranking combination exists; no excluding function or AUROC script is in the repository.
• Prompts. Paper: the keyword, matching and aggregation prompts are in Supplementary Tables 2-5 p. 31-35, which are not in our PDF. The prompts exist only in code, so the paper's exact wording cannot be checked against code beyond what the repo shows.
• Data. The repository does not ship dataset/trial_info.json or the TREC 2021 and 2022 corpora (corpus.jsonl); README gives NCBI FTP download commands for them README.md README.md. The SIGIR corpus is included.

Pitfalls

• Cohort scope. The 87.3% criterion accuracy comes from manual checks on the SIGIR cohort only p. 12. The paper reports no manual criterion check for the TREC cohorts, so do not generalise the figure.
• Pooled labels. Retrieval is measured against judged trials pooled for each cohort. The paper notes that eligible trials outside that labelled set are counted as irrelevant if retrieved, so the metrics may understate performance p. 9.
• Zero division. Eq. 4-11 divide by M' or N'. The paper does not say what happens when every criterion is not applicable. The code avoids division for counts with a small epsilon rank_results.py; implement a clear fallback and document it.
• Model drift. Results depend on GPT-4 (0613) and GPT-3.5 (0613), which are closed models. Temperature 0 is set, but outputs are not guaranteed identical across endpoints or dates p. 31. Log the model string and date with every run.
• Cost. The matching step makes two LLM calls per patient-trial pair p. 33. The analysis considers the top 500 candidates per patient p. 16, so a single patient under that setting costs up to 1,000 matching calls plus one aggregation call. Computed from the stated numbers; the paper does not give a total cost.
• Explanation vs label. The explanation is generated before the label, which the paper motivates with chain-of-thought prompting p. 33. Its error analysis attributes 30.7% of the errors to incorrect reasoning p. 16, so the stated reasoning can be wrong even when a label is right. Treat the explanation as an aid for human review, not as evidence.
• Label semantics. The prompt says "not enough information" should be used as little as possible, because a fact the note does not mention is assumed not true TrialGPT.py. The paper attributes 26.9% of errors to ambiguous or overlapping label definitions p. 16.
• Ranking metric mismatch. Table 2 mixes two paper numbers for the same row (P@10 0.6699 in text, 0.6688 in table) p. 21-22. Use the table value and record the discrepancy if you reproduce it.
• Excluding is hard. The best baseline's AUROC for excluding is 0.6176, close to random p. 22. TrialGPT's excluding AUROC is 0.7979, but the excluding-specific score is not in the repo.
• Synthetic patients. The 183 patients are described as synthetic p. 1-2; the paper names EHR data, longer records, labs and imaging as unevaluated p. 27. Geolocation and recruitment status are not modelled p. 9 p. 27.
• User study scope. Six oncology trials, six vignettes, two annotators, 36 pairs p. 23 p. 40. The authors call the study limited in scope p. 28. The 35.3 s vs 61.5 s figures are for those pairs, not for real screening.
• Human oversight. The paper says the system is not meant to replace recruiters and that experts should stay in the loop p. 27.

Lineage

• Builds on: reciprocal rank fusion p. 32, ref. 52; BM25 p. 31, ref. 50; MedCPT dense retriever p. 31, ref. 51; chain-of-thought prompting p. 33, ref. 53.
• Compared against: SciFive, BioBERT, PubMedBERT, SapBERT and BioLinkBERT, fine-tuned on biomedical NLI datasets (MedNLI, MNLI, SNLI, SciNLI, SciTail, STSB) p. 21-22, tab. 2. Earlier patient-trial matching work cited in the introduction includes DeepEnroll and COMPOSE p. 43, refs. 15-16.
• Superseded by: not known from content already in this wiki.