wikillm

Paper · 2023-06-30 · cs.AI, cs.CY, cs.LG

Transformers in Healthcare: A Survey

Survey of Transformer models in healthcare: clinical NLP, EHR, medical imaging, social media, bio-physical signals and biomolecules, plus computational cost, interpretability, bias, alignment and privacy. Built from 298 studies after a PRISMA-style search. It runs no experiments: its numbers are screening counts and figures cited from other work.

Attention mechanismSelf-attentionPositional encodingVision Transformer (ViT) backbonessurveyhealthcaretransformer

arXivPDFMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Transformers are a neural network design built around attention. Each piece of an input looks at the other pieces and decides how much each one matters. The design was developed for language tasks and now underlies many language and image models.

This survey asks where that design has been applied in healthcare. It groups the work by kind of data: clinical text, structured patient records, medical images, social media posts about health, heart and brain signals, and molecules such as DNA, proteins and drugs. For each group it lists the models used and the tasks they address.

The authors also look beyond accuracy at the costs and risks: how much compute and energy large models need, whether their decisions can be explained, whether they treat patient groups fairly, and how they handle privacy and data sharing. The survey runs no experiments of its own. It collects published work found through a structured literature search.

TL;DR

A survey, not a method. It maps Transformer applications across six healthcare data types, starting from 2154 records and ending with 298 studies after full-text screening fig. 1. Use it as a map of which architectures were tried on which data, not as evidence that any of them works. Its architecture section gives usable equations for attention and the encoder block, but several are ambiguous (see Pitfalls).

Problem

Transformer work in healthcare is spread across many subfields, data types and papers, and the costs and risks of the approach are discussed separately from the models. The authors aim to give a single overview of Transformer models across modalities and to cover pre-training strategies that address scarce annotated healthcare data §1.

Method

The survey has two parts: a literature search and a summary of the architecture.

Literature search. The authors searched Google Scholar and PubMed §2. They limited the search to studies published after 2017, the year the Transformer was introduced §2 §1. Keywords were combined with AND and OR, and each query was restricted to the healthcare domain using the terms "health", "medical" or "clinical" §2. Table 1 lists the exact queries tab. 1. Harzing's Publish or Perish retrieved the studies and Covidence ran the PRISMA steps §2. The screening funnel and exclusion criteria are shown in fig. 1.

Architecture summary. The survey restates the original Transformer, then BERT and the Vision Transformer (ViT) §3 §4. Its equations are reproduced below, with the paper's own numbering.

  1. Attention. Queries, keys and values are linear projections of the input §3.1 eq. 1-3. Scaled dot-product attention applies a softmax to the scaled query-key scores and uses that to mix the values §3.1 eq. 4.
  2. Attention variants. Self-attention takes queries, keys and values from one sequence §3.2.1. Masked self-attention adds an upper-triangular mask so that a position cannot read later positions §3.2.2 eq. 5-6. Cross-attention takes queries from one sequence and keys and values from another §3.2.3.
  3. Multi-head attention. Several heads run in parallel, each with its own projection matrices; their outputs are concatenated and projected back §3.2.4 eq. 7-8.
  4. Feed-forward network. Two linear layers with ReLU, expanding from the model dimension to a larger feed-forward dimension and back §3.3 eq. 9.
  5. Residuals and layer normalisation. The survey states these are used around each sub-layer §3.4 eq. 11-15.
  6. Positional encoding. Sine and cosine functions at different frequencies, added to the token embeddings, with the same dimension as the embedding §3.5 eq. 10.
  7. Encoder and decoder blocks. The encoder is self-attention followed by a feed-forward layer; the decoder adds masked self-attention and cross-attention §3.6 eq. 11-15.
  8. Three usage patterns: encoder-decoder, encoder-only and decoder-only §3.8.
  9. Cost. Self-attention is O(n² · d) in sequence length n and model dimension d §3.7.

Key equations, reproduced from the paper:

Attention(Q, K, V) = softmax(\alpha\, Q K^{T})\, V eq. 4

PE_{(pos,i)} = \begin{cases} \sin(pos \cdot \omega_k) & i = 2k \\ \cos(pos \cdot \omega_k) & i = 2k+1 \end{cases}, \quad \omega_k = \frac{1}{10000^{2k/d}}, \quad k = 1, \dots, \frac{d}{2} eq. 10

Pseudocode for the architecture pieces the survey describes, written for this card. Each step cites the equation or section it follows.

Pseudocode
# Scaled dot-product self-attention  [eq. 1-4] [§3.1]
# Input:  inputs (n, d); w_q, w_k (d, d_k); w_v (d, d_v)
# Output: attended values (n, d_v)

def self_attention(inputs, w_q, w_k, w_v):
    # Project every token into a query, a key and a value  [eq. 1-3]
    queries = inputs @ w_q                     # (n, d_k)
    keys = inputs @ w_k                        # (n, d_k)
    values = inputs @ w_v                      # (n, d_v)
    # Score each query against each key, scaled by alpha  [eq. 4] [§3.1]
    # The text says alpha is typically 1 / sqrt(d_k)
    scores = alpha * (queries @ keys.T)        # (n, n)
    # Normalise each row, then mix the values with those weights  [eq. 4]
    weights = softmax(scores, axis=-1)         # (n, n)
    return weights @ values                    # (n, d_v)
Pseudocode
# Causal mask and masked attention  [eq. 5-6] [§3.2.2]

def causal_mask(n):
    # Entry (i, j) is minus infinity when i < j, otherwise 0  [eq. 5]
    mask = zeros(n, n)
    for i in range(n):
        for j in range(n):
            if i < j:
                mask[i, j] = -inf
    return mask

def masked_attention(queries, keys, values, mask):
    # Add the mask to the scaled scores before the softmax  [eq. 6]
    scores = alpha * (queries @ keys.T) + mask
    weights = softmax(scores, axis=-1)
    return weights @ values
Pseudocode
# Multi-head self-attention  [eq. 7-8] [§3.2.4]
# Input: inputs (n, d); one weight set per head
# Output: (n, d)

def multi_head_attention(inputs, w_q, w_k, w_v, w_o):
    heads = []
    for i in range(len(w_q)):
        # Each head runs its own attention with its own projections  [eq. 7]
        heads.append(self_attention(inputs, w_q[i], w_k[i], w_v[i]))
    # Concatenate the heads and map back to the model width  [eq. 8]
    return concat(heads, axis=-1) @ w_o        # (n, d)
Pseudocode
# Position-wise feed-forward network  [eq. 9] [§3.3]

def feed_forward(hidden, w1, b1, w2, b2):
    # Expand to width d_f, apply ReLU, project back to d_m  [eq. 9]
    expanded = relu(hidden @ w1 + b1)          # (n, d_f)
    return expanded @ w2 + b2                  # (n, d_m)
Pseudocode
# Encoder block as the paper writes it  [eq. 11-12] [§3.6]
# Layer norm is applied to the sub-layer output, then the input is added

def encoder_block(inputs, attn_w, ffn_w):
    # Attention sub-layer  [eq. 11]
    mixed = layer_norm(multi_head_attention(inputs, *attn_w)) + inputs
    # Feed-forward sub-layer  [eq. 12]
    return layer_norm(feed_forward(mixed, *ffn_w)) + mixed
Pseudocode
# Sinusoidal positional encoding  [eq. 10] [§3.5]
# Output: table (n, d), added to the token embeddings

def positional_encoding(n, d):
    table = zeros(n, d)
    for pos in range(n):
        for i in range(d):
            # Frequency index k, taken from i; the paper writes k = 1..d/2
            # (see Pitfalls on the start index)  [eq. 10]
            k = i // 2
            omega = 1 / 10000 ** (2 * k / d)
            # Even i takes sine, odd i takes cosine  [eq. 10]
            if i % 2 == 0:
                table[pos, i] = sin(pos * omega)
            else:
                table[pos, i] = cos(pos * omega)
    return table

Hyperparameters

The survey is not an experiment, so it reports almost no training settings. These are the values stated for the architectures it describes.

NameValueWhereNotes
attention heads h, original Transformer8§3.2.4
encoder blocks, original Transformer6§3.6
decoder blocks, original Transformer6§3.6
attention scale alphatypically 1 / sqrt(d_k)§3.1the text says "typically"; the equation leaves alpha open
d_q = d_k = d_v in self-attentionequal§3.2.1
feed-forward dimension d_f"usually larger than d_m"§3.3no value stated
positional-encoding dimensionequal to the embedding dimension§3.5
mask Mminus infinity above the diagonal, 0 otherwiseeq. 5
ViT patch size16×16 or 32×32, "generally"§4.2
ViT classification headone hidden layer§4.2width not stated
BERT pre-trainingmasked language modelling and next-sentence prediction§4.1masking fraction not stated; next sentence is the true one in 50% of pairs
search windowstudies from 2017 onward§2search date not stated
search enginesGoogle Scholar and PubMed§2
learning rate, batch size, optimisernot stated—the survey gives no training settings for the surveyed models

Results

No experiments of its own. The survey reports a screening funnel and summary tables; numbers it quotes elsewhere are cited to their sources.

Screening funnel, as printed in fig. 1:

StageCountSource
Identified (Google Scholar, PubMed)2154fig. 1
Duplicates removed354fig. 1
After duplicate removal, as printed1600fig. 1 (see Pitfalls)
After title and abstract screening554fig. 1
After full-text screening298fig. 1

Summary tables, not results of the authors:

  • Table 2 lists clinical and biomedical Transformer models with their tasks, datasets and base architecture tab. 2.
  • Table 3 lists transformer-based medical image segmentation models with datasets, tasks and modalities tab. 3.

Figures and numbers quoted from other work (author-cited, not verified here):

QuantityValueSource
CO2 for training a large deep learning model626,000 lbs§13.2, citing [465]
GPT-3 training cost and compute time$4.6 million; 355 years on one Tesla V100§13.3, citing [25] and [469]
PaLM training energy103,500 kWh over 60 days§13.3, citing [470]
Quantisation to 2 bitsabout 2.3% performance loss; 13× smaller weights, 4× smaller embeddings and activations§13.3.2, citing [476]

Paper vs code

No official code.

The survey proposes no model and has no implementation. The paper links no repository or project page in the text I read (pages 1–40, through the start of the conclusion). I did not read the reference list. The arXiv record in raw/arxiv/2307.00067.json lists no comment or code link. A GitHub repository search for "transformers healthcare survey" with author name Nerella returned zero results.

Pitfalls

  • No experiments. Every number here is either a screening count or a figure quoted from another paper. Nothing in the survey is reproducible in the sense an implementer needs. Do not read the summary tables as a leaderboard: the tasks, datasets and metrics differ across rows.
  • PRISMA arithmetic. The figure shows 2154 identified and 354 duplicates removed, which leaves 1800, but it prints 1600 after duplicate removal fig. 1. The later counts (554, 298) are taken as printed. The paper does not explain the gap.
  • Category count. Section 2 says the search had six categories and lists clinical NLP, EHR, social media, medical imaging, biomolecules and bio-physical signals §2. Table 1 has seven rows and adds critical care tab. 1; the survey later has a critical care section §9.
  • Search date and scope not stated. The search window starts in 2017, but the paper gives no end date and no language restriction §2. Results will age quickly.
  • Citation numbers are unreliable as a map. The paper cites [1] for healthcare data growth §1 and for the Vision Transformer §4.2. ViT also appears as [22] in the Figure 7 caption and as [248] in §8.1.1 and §8.2. Do not map a number to a reference without the reference list.
  • Equation 10 start index. The paper writes k = 1..d/2 and assigns sine when i = 2k and cosine when i = 2k+1 eq. 10. Read literally, i = 0 and i = 1 have no k. The pseudocode above uses k = i // 2, which starts at 0. The paper does not say which is intended; decide the start index explicitly before implementing.
  • Equation 11–12 layer-norm placement. Equations 11–12 put the layer norm on the sub-layer output and then add the input eq. 11-12. Figure 5 labels the blocks "Add & Norm" fig. 5, which reads the other way round. The paper does not say which is meant; check the original architecture before implementing.
  • Equation 14 reuses the symbol Y_int. The same name appears on both sides of the cross-attention residual eq. 14. The natural reading is that the right-hand term is the block input, but the paper does not state this.
  • Attention scale. Equation 4 includes an alpha without defining it; the text says alpha is typically 1 / sqrt(d_k) §3.1. Use the text value and say so in code.
  • Cost figures are dated and cited. The energy and cost figures §13.2 §13.3 come from earlier papers and models, not from this survey's measurements. Compute and carbon numbers for newer models will differ.
  • Interpretability claim is cited. The statement that attention is often fragmented and not a robust explanation comes from reference [461] §13.1; the survey does not test it.

Lineage

  • Builds on: the original Transformer (Vaswani et al., 2017, cited as reference [2]) §1 §3.6; BERT (reference [21]) §4.1; Vision Transformer (ViT, cited as [1] in §4.2 and [22] in Figure 7) §4.2.
  • Compared against: not applicable; the survey compares no models with each other.
  • Superseded by: not stated in this card.

Source and license

Adapted from “Transformers in Healthcare: A Survey” by Subhash Nerella, Sabyasachi Bandyopadhyay, Jiaqing Zhang, Miguel Contreras, Scott Siegel, Aysegul Bumin, Brandon Silva, Jessica Sena, Benjamin Shickel, Azra Bihorac, Kia Khezeli, Parisa Rashidi (arXiv:2307.00067), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.