Large Language Models are Powerful Electronic Health Record Encoders

In plain words

Hospitals record a patient's history as thousands of coded entries: diagnoses, lab tests, drugs, procedures. Usually a model for predicting outcomes is built on a special-purpose network trained on a large private record collection. This paper asks whether a general-purpose language model, the kind used for search and text similarity, can do the same job.

The trick is to turn the codes into ordinary sentences. Each code is replaced by its readable name, and the patient's record becomes a plain list of these names, with the most recent entry for each code kept. A text embedding model reads that list and produces one vector per patient and per prediction question. A simple classifier is then trained on those vectors, one per clinical question.

On a standard benchmark of 15 prediction tasks, this matched the specialised record model. On a second, external cohort the general model did slightly better overall. The price is compute: the general model takes many hours to process the data, where the specialised one takes minutes.

TL;DR

• Serialise each patient's EHR as a newline list of code descriptions (newest occurrence per code), truncated to 8192 tokens.
• Prepend a task-specific instruction and take the last-token hidden state of Qwen3-Embedding-8B (4096-d) as the patient embedding.
• Train one L2-regularised logistic regression per task on the embeddings; no fine-tuning of the encoder in the main setting.
• Author-reported: EHRSHOT macro-AUROC 0.769 (all training data), equal to CLMBR-T-Base 0.769; UK Biobank macro-AUROC 0.751 vs 0.736 §2.2 tab. 3 tab. 5.
• Trade-off: about 22 h of encoding on 8 H200 GPUs vs 6 min for CLMBR-T-Base tab. S12.

Problem

Specialised EHR foundation models are pretrained on large private record collections and use a fixed code vocabulary. Both limit portability: a model trained at one site cannot map codes it has never seen, and mapping another site's codes is a large effort (about two weeks for the UK Biobank cohort in this study) §1 §4.10. On the UK Biobank cohort, only 16% of the 50,702 distinct codes (25% of clinical events) could be mapped to the vocabulary of CLMBR-T-Base §4.10 §3.

Earlier LLM-based EHR studies often used short context windows or private cohorts, which limits longitudinal coverage and external validity §1. Encoder-only biomedical models need chunking for long records §3.

The paper's assumption is that a long-context, instruction-tuned LLM embedding model can read long code lists directly without a site-specific mapping, and that a frozen encoder with a linear head is enough §1 §2.6.

Method

Overview. The paper does not propose a new architecture. It evaluates a fixed pipeline: serialise, embed, classify §1 fig. 1.

1. Cohort and labels. EHRSHOT (6,739 patients, 15 tasks in 4 groups) with its predefined splits §4.1 tab. 2. For the external check, UK Biobank with 387,464 patients and 25 tasks (mortality, one-year hospitalisation, and 23 incident diagnoses) §4.10 tab. S1.
2. Labels per prompt. Each (patient, task, prediction time) gets its own serialisation, because the task instruction changes the embedding. The authors re-encoded these instances instead of reusing one encoding per patient and time, giving 1,161,412 EHRs instead of 406,379 §4.5.
3. Serialisation §4.2. Keep all events up to and including the label time. Map each event's code (ontology/code) to text. Keep only the most recent occurrence of each code. Write one line per code: the description, then value and unit in brackets when present. Round floats to two decimals. The default list adds no date or time stamps §2.4 §4.2 fig. 2 tab. 6.
4. Truncation. Cap each serialisation at 8192 tokens (about 32,000 characters) §4.2 §2.1.
5. Instruction. Prepend a shared prefix to a task-specific question, for example "has the patient anemia" tab. S3 §4.5.
6. Embedding §4.4. Run the model and take the last-layer hidden state of the final token. The authors used each model's default embedding setup. For LLM2Vec the paper mean-pools instead, excluding instruction tokens §4.4.
7. Classifier §4.4. Train a logistic regression per task on training embeddings. Select hyperparameters on the validation split. The same head is used for every embedding model, to isolate representation quality §2.1 §4.4.
8. Evaluation. EHRSHOT few-shot protocol, with k in {1, 2, 4, 8, 12, 16, 24, 32, 48, 64, 128} positive and k negative training examples per task §4.1. Full-data results use all training and validation examples §2.2.

# Serialise one patient at one prediction time  [§4.2]
# Input:  events (code, start, value, unit); label_time
# Output: one text string, one line per code

def serialize(events, label_time):
    # Keep events up to and including the label time  [§4.2]
    past = [e for e in events if e.start <= label_time]
    # Newest first  [code: ehrshot/serialization/ehr_simple_serializer.py#L202]
    past.sort(key=lambda e: e.start, reverse=True)
    # Keep only the newest occurrence of each code  [§4.2]
    newest = first_occurrence_per_code(past)
    # Description, then [unit] and value when present; floats to 2 dp  [§4.2]
    lines = [describe(e) for e in newest]
    return "\n".join(lines)

# Build the model input, then embed it  [§4.4] [§4.5] [tab. S3]
# Shapes: tokens (B, T) left-padded; hidden (B, T, 4096)

def build_input(task_question, text):
    # Shared prefix plus the task question  [tab. S3]
    # [code: ehrshot/llm_featurizer.py#L364-L368]
    instruction = PREFIX + " " + task_question
    # Qwen3 query template
    # [code: ehrshot/serialization/text_encoder.py#L229-L232]
    return "Instruct: " + instruction + "\nQuery:\n" + text


def embed(batch_texts, model, tokenizer):
    # Tokenise, truncating at 8192 tokens  [§4.2] [code: text_encoder.py#L216]
    tokens = tokenizer(
        batch_texts,
        max_length=8192,
        truncation=True,
        padding=True,
    )
    hidden = model(**tokens).last_hidden_state
    # Last position; valid because of left padding
    # [code: ehrshot/serialization/text_encoder.py#L201-L209]
    vector = hidden[:, -1]
    # L2-normalise each vector (code only; not in the paper)
    # [code: ehrshot/serialization/text_encoder.py#L219]
    return vector / norm(vector, axis=1)   # (B, 4096)

# Fit one head per task; choose C on validation  [§4.4] [§4.1]
# Input: X_train, y_train, X_val, y_val  (embeddings, labels)

def fit_task_head(X_train, y_train, X_val, y_val):
    # Max-abs scaling, fitted on training data only
    # [code: ehrshot/7_eval.py#L104-L107]
    scaler = MaxAbsScaler().fit(X_train)
    X_train = scaler.transform(X_train)
    X_val = scaler.transform(X_val)
    # Grid of inverse regularisation strengths
    # [code: ehrshot/utils.py#L237-L240]
    grid = [
        1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1,
        1, 1e2, 1e3, 1e4, 1e5, 1e6,
    ]
    # Pick C by validation AUROC; refit on training data only
    # [code: ehrshot/7_eval.py#L41-L55]
    return select_c_by_val_auroc(grid, X_train, y_train, X_val, y_val)

Decoder variant §2.6 §4.6. The paper also scores a decoder, Qwen3-8B, with a prompt that asks for a single token, "Yes" or "No". The probability of each answer is the total mass over its single-token spellings, normalised over the two answers. It is either used with 0, 2, 4 or 6 in-context examples, or LoRA-tuned to emit Yes or No.

# Score a decoder answer as P(Yes) versus P(No)  [§4.6]
# Input: prompt_text (instruction + serialised EHR)

def p_yes(prompt_text, model):
    # Next-token distribution after the prompt  [§4.6]
    probs = softmax(model(prompt_text).next_token_logits)
    # Sum over capitalisation and punctuation variants  [§4.6]
    mass_yes = sum(probs[t] for t in YES_TOKEN_VARIANTS)
    mass_no = sum(probs[t] for t in NO_TOKEN_VARIANTS)
    # Normalise to a two-way probability  [§4.6]
    return mass_yes / (mass_yes + mass_no)

Fine-tuning §2.6 §4.6. The encoder variant adds LoRA adapters on the attention and MLP projections of Qwen3-Embedding-8B, with a dropout layer and a linear head trained with cross-entropy. The decoder variant adds the same kind of adapters to Qwen3-8B. Hyperparameters were tuned on two task groups at k in {8, 16}, then fixed for the full matrix §4.6 tab. S13.

Count baseline §4.4. Counts of each medical concept per patient, with ontology expansion. Three extensions add string values, numeric values in deciles, and time bins (0–30, 30–180, 180–365, 365+ days). A gradient-boosted model is the main classifier for counts; a logistic regression is reported too §4.4 tab. S6.

Statistics §4.11. Per-task AUROC differences use a paired patient-level bootstrap with 10,000 replicates and two-sided p-values. Holm correction runs over 45 hypotheses per shot setting on EHRSHOT, and over 75 (8-shot, all data) and 60 (64-shot) on UK Biobank §4.11.

No numbered equations or algorithms appear in the paper. The steps above are described in §4.2–§4.6, and the reader must rely on those sections.

Hyperparameters

Results

Author-reported. Setup: EHRSHOT (6,739 patients, 15 tasks, 4 groups) with predefined splits; UK Biobank external validation (387,464 patients, 25 tasks) §2.1 §4.1 §4.10. Baselines: CLMBR-T-Base (an EHR foundation model trained on Stanford records) and a count-based GBM §4.4. Hardware: 8-GPU NVIDIA H200 cluster §4.8. Numbers are macro-averaged AUROC with approximate 95% CIs, all training data unless noted tab. 3.

EHRSHOT and UK Biobank, macro-AUROC (all training data)

EHRSHOT per task group, macro-AUROC with CIs in the source table (all data)

Source: tab. 3. Per-group CIs are in the source table.

Task-level significance (win / tie / loss for Qwen3-Emb-8B + LR) tab. 4

Counts are over 15 EHRSHOT tasks and 25 UK Biobank tasks §4.11. The text also says UK Biobank gains are explained partly by vocabulary coverage: restricting to CLMBR-mappable codes gives 2 wins over CLMBR-T-Base instead of 6 §2.2 tab. 4.

Serialisation variants, Qwen3-Emb-8B, macro-AUROC tab. 6 tab. S7

Context length, EHRSHOT macro-AUROC tab. S10

Time windows are shown in fig. 7 and tab. S11. Qwen3-Emb-8B peaks at one year; Qwen2-Emb-7B at one month; Llama3.1-LLM2Vec-8B at one week §2.5. A one-hour window drops performance for every model §2.5.

Few-shot and fine-tuning, macro-AUROC at k = 128 §2.6 (appendix table labelled tab:ehrshot_k128_encoder_decoder)

For the decoder with in-context examples, the best result was 0.636 at 2 shots and 0.611 at 0 shots §2.6. The LoRA decoder reaches 0.708 at k = 64 and the LoRA encoder 0.691 §2.6. The authors conclude a frozen embedding with a simple classifier is the default §3.

Compute tab. S12 (8-GPU H200 cluster, EHRSHOT encoding time)

On UK Biobank, one task's encoding took 4:46:34 for Qwen3-Emb-8B, 3:49:21 for Qwen2-Emb-7B, and 3:55:12 for Llama3.1-LLM2Vec-8B §4.8.

Figures. fig. 3 shows scaling with model size: Qwen and DeBERTa change little, larger BERT variants do worse, and CLMBR-T-Base is the most parameter-efficient. fig. 4 and fig. 5 show few-shot curves on EHRSHOT and UK Biobank. fig. 6 shows serialisation ablations. fig. 8 compares encoder and decoder fine-tuning.

Paper vs code

Code: stefanhgm/ehrshot-benchmark at commit 2fddc1e166cb9a6951e9f8843b440ad40376c00d, Apache-2.0 (LICENSE file), linked by the paper in §4.1 and its Code Availability section. Under our rule a repository the paper links counts as official, so I did not check the owner's identity further. I read the serialisation, encoder, classifier and fine-tuning scripts. I did not read the UK Biobank pipeline in UKB_validation/ line by line, so UK Biobank code claims are not checked here.

• Embedding normalisation. The paper does not mention L2 normalisation. The code L2-normalises the Qwen embeddings before the classifier text_encoder.py. This changes what the classifier's C means, so it matters for reproduction.
• Scaling before the classifier. The paper says a logistic regression is trained on the embeddings. The code first applies max-abs scaling, fitted on training data only 7_eval.py.
• Query template. The paper says the instruction is prepended. For Qwen3 the code wraps it as "Instruct: " then a newline, "Query:", a newline, then the serialisation text_encoder.py. The prefix and the operational, lab and diagnosis questions match the paper's table tab. S3 task_to_instructions_list.json. The code also has per-finding chest X-ray questions for its 14 sub-tasks; the paper gives one chest X-ray question tab. S3 §4.6.
• Event order. The paper does not state the order of the list. The code sorts events newest first before deduplicating ehr_simple_serializer.py. The default strategy name is unique_codes_list_recent_8k run_experiments.sh.
• Precision. The paper does not state the numeric format. The code loads Qwen3-Embedding-8B in float16 with FlashAttention 2 text_encoder.py.
• Batch size. The paper says the effective batch size is 8 §4.6. The encoder script defaults to min(k, 8) when unset 10a_fit_and_eval_encoder.py. The shipped configs set 8 best_params_encoder.json best_params_decoder.json, so the paper's setting holds for the published runs.
• Missing ontology file. The code reads ehrshot/custom_ontologies/cpt4.csv when the featuriser is constructed llm_featurizer.py. That file is not in the repository; the ontology readme links an external gist instead Readme.md. Without it, construction fails. The paper says manual mapping files were added for CPT4, ICD-10-PCS and CVX §4.2.
• Decoder prompt. The paper calls the decoder prompt a chat-style prompt with the Yes/No instruction §4.6. The code puts the instruction in a "Question:" block with the Yes/No constraint 10b_fit_and_eval_decoder.py.
• LLM2Vec code. The authors removed a word-boundary token-limit routine from LLM2Vec, with "minimal effect" per the paper §4.8. This is a change to the reference implementation, and it is stated in the paper.
• Hyperparameter grid. Code grid for LR C is 1e-6 to 1e6 with validation AUROC selection utils.py 7_eval.py. The paper does not give the grid.

LoRA settings match the paper: the shipped encoder config has learning rate 5e-5, rank 8, dropout 0.1, and the decoder config has 2e-4, rank 8, dropout 0.05 best_params_encoder.json best_params_decoder.json, as in tab. S13. Both use per-device batch 4 with effective batch 8 tuning_grid.yaml.

Pitfalls

• Ambiguous replicate protocol. The paper calls the few-shot runs "five bootstrapped runs using different seeds" fig. 4 and also "five-fold cross-validation" §4.11. The text does not settle which one was used for each reported value.
• Vocabulary coverage is confounded. On UK Biobank, Qwen3-Emb-8B gains over CLMBR-T-Base come partly from broader code coverage (16% mapped) and partly from better generalisation §2.2. The restricted run is intermediate, so neither reading is clean.
• Long-context dependence. Only Qwen3-Emb-8B handled 8192-token lists robustly. Qwen2-Emb-7B and LLM2Vec drop at long contexts §2.5 tab. S10. Do not swap in another embedding model at 8192 tokens without re-checking.
• Truncation drops oldest codes. Lists are newest first, so truncation removes the oldest entries. The code does not set the truncation side explicitly; it uses the tokenizer default text_encoder.py. Check the side on your tokenizer.
• Cost. Qwen3-Emb-8B took about 22 hours for EHRSHOT on 8 H200 GPUs, against 6 minutes for CLMBR-T-Base tab. S12. Memory cost is also much higher §3.
• Instructions matter. A generic prompt lowers performance, and no prompt lowers it further. The largest drop is on lab prediction §2.4 fig. 6.
• Count baseline is strong with data. On all data the count baseline matches or beats the LLM embedding on several tasks, so do not read the result as LLM embeddings always winning §2.2 tab. 4.
• Fine-tuning tuned narrowly. LoRA tuning used only two task groups and k in {8, 16}, one shared setting per family §4.6. The paper says full per-task tuning was not feasible.
• Decoder in-context learning capped. ICL stops at 6 examples, with a 4096-token input cap §4.6. Results beyond 6 shots are not reported.
• Data access. EHRSHOT is under a gated research data use agreement §4.1 [Data Availability]. UK Biobank requires an application [Data Availability]. Reproduction needs both.
• Sample size. Labels are per (patient, task, prediction time). Many patients contribute several labels, so counts differ by task §4.1 tab. 2.
• Equation and algorithm numbers. The paper has none. Implementers must work from §4.2 to §4.6.

Lineage

• Builds on: CLMBR-T-Base (steinberg_language_2021 in the paper), the EHRSHOT benchmark (wornow_ehrshot_2023), and the MEME method (lee_clinical_2025), which is compared as a baseline §4.4 §4.7.
• Uses: Qwen3-Embedding-8B (yang_qwen3_2025; zhang_qwen3_2025), GTE-Qwen2-7B-Instruct, and LLM2Vec-Llama-3.1-8B-Instruct §4.4.
• Compared against: BioClinicalBERT and MedBERT chunked baselines, the count-based GBM from EHRSHOT, and CLMBR-T-Base §4.4 tab. 3.
• Related work the paper cites for LLM-based EHR prediction: Gao et al. (embedding comparison on MIMIC-III), GRASP (kirchler_large_2026), DeLLiriuM (contreras_dellirium_2024) §4.7.
• Superseded by: not known from content already in this wiki.