Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

In plain words

Transformers, the model type behind modern chatbots, were first built for text. Since then they have also been applied to images, where they look at relationships across the whole picture instead of only small neighbourhoods, as a convolutional network does. This paper is a large survey of how that idea has been used in medical imaging: reading X-rays and scans, outlining organs and tumours, reconstructing images from less data, translating one scan type into another, finding objects, lining up two images, and writing reports.

For each task the authors group the published models into families, list what each one does well and badly, and collect the benchmark numbers that papers reported. The paper does not propose a new model of its own. It is most useful as a map of the field and as a starting point for choosing an architecture.

TL;DR

• A survey of Vision Transformers in medical image analysis, covering seven tasks and more than 200 papers according to the introduction §1.
• Each task gets a taxonomy, and the review lists strengths and weaknesses per model fig. 1 §3 to §9.
• The only numeric comparisons are the benchmark tables for segmentation tab. 6 tab. 7, compiled by the authors and not re-run here.
• The open problems named are cost of attention and pre-training, explainability, video, registration, and software tooling §10.

Problem

The review says CNNs have a limited receptive field, so they struggle with long-distance dependencies, and their convolution kernels use fixed weights at inference §1. Attention adds input-dependent weighting and global context, but its cost grows quadratically with the number of positions, which is a burden on dense, high-resolution medical images §1. The review also argues that clinical reading varies between readers and is slow and expensive, which motivates automated analysis §2.3.

Method

This is a literature review. The authors searched DBLP, Google Scholar and Arxiv Sanity Preserver with keyword combinations built from "transformer", "medical", "image" and each task name §1, Search Strategy. They say they kept the papers they judged most novel, significant and first of their kind, and that the review may have left out other important work §1. The search date range is not stated.

The review organises the field in two ways.

1. By task, in these sections: classification §3, segmentation §4, reconstruction §5, synthesis §6, detection §7, registration §8, report generation §9. Each section proposes its own taxonomy; the segmentation discussion summarises the reviewed models and their tables §4 discussion.
2. By where the Transformer sits in the network §2.2:
   • Pure Transformer: an encoder-decoder built only from Transformer blocks, such as Swin-Unet and TransDeepLab §2.2.
   • Hybrid: a CNN with a Transformer in the encoder (to add global context), in the decoder, or in the skip connections (to fuse low- and high-level features). TransUNet and HiFormer are the review's examples §2.2.
   • Other: the remaining placements, such as a Transformer in the bottleneck §2.2.

The background explains the ViT pipeline §2.1 fig. 2: an image is cut into fixed-size patches that are flattened and linearly projected into patch embeddings. Positional embeddings are added, and a learnable class token is appended. A stack of encoder blocks follows, each with multi-head self-attention and an MLP, both pre-normalised with LayerNorm and wrapped in residual connections. A final MLP head produces the prediction §2.1.

Self-attention projects each token into a query, key and value, then forms weights from scaled query-key dot products and applies them to the values §2.1 eq. 1 eq. 2. Multi-head attention runs several such heads in parallel, concatenates them and applies an output projection eq. 3. The review states that the original paper used h = 8 heads §2.1.

# Scaled dot-product self-attention  [§2.1] [eq. 1] [eq. 2]
# Input:  tokens (N, D); learned w_q, w_k, w_v with shape (D, key_dim)
# Output: attended tokens (N, key_dim)

def self_attention(tokens, w_q, w_k, w_v):
    # Project every token to a query, key and value  [eq. 1]
    queries = tokens @ w_q                      # (N, key_dim)
    keys = tokens @ w_k                         # (N, key_dim)
    values = tokens @ w_v                       # (N, key_dim)
    # Score all pairs, scale by sqrt(key_dim), normalise each row  [eq. 2]
    scores = queries @ keys.T / sqrt(key_dim)   # (N, N)
    weights = softmax(scores, axis=-1)
    # Weighted sum of values gives the output
    return weights @ values

Multi-head attention eq. 3 is the same operation run h times on separate projections, then concatenated and mapped by an output matrix W^O. The review does not give the value of key_dim for any reviewed model.

Discussion of efficiency in §10 lists several ways to reduce the attention cost: windowed attention (Swin-style), attention across channels instead of tokens (XCiT-style), reordering the query-key-value products, linear attention on a region of interest, and spatial reduction of the keys (ESA). The review gives complexity expressions for some of them, for example O(dn²) → O(d²n) for an efficient attention variant and O(n²/R) for ESA with reduction ratio R §10. These are quoted from the cited works, not re-derived.

Hyperparameters

The review trains no models, so it sets no training hyperparameters. The values it does state:

Results

Author-compiled comparisons, not re-run. The review does not say which runs the numbers come from, and it gives no training protocol. Setup: each row is a published model, and each table lists parameters and FLOPs where the source paper reported them. A dash means not reported.

Synapse multi-organ segmentation, ordered by Dice tab. 6:

Skin lesion segmentation on ISIC 2017, ISIC 2018 and PH² tab. 7:

Other tasks: the review gives per-task comparison tables for classification, reconstruction, synthesis, detection, registration and report generation, but this card has not transcribed them. Read the source for those numbers.

Headline numbers, as the review reports them:

• Synapse: MERIT has the best Dice (84.90) and the best HD95 (13.22) in tab. 6.
• ISIC 2017 and 2018: D-LKA Net has the best DSC (0.9254 and 0.9177) in tab. 7.
• Discussion of inference cost in §4 gives per-model times and GPU memory (for example DAE-Former 46.01 ms and 6.42 GB, MERIT 93.13 ms and 7.78 GB). These are in the text only, not in a table.

Paper vs code

No official code. The abstract and the introduction link https://github.com/mindflow-institue/Awesome-Transformer. That repository is an index of the papers the review covers, with links to their implementations, not code for the review's own method. The review presents no method, so there is nothing to reproduce. Searched: the links in the abstract and introduction only. No GitHub search was run.

Pitfalls

• Dice mismatch. The segmentation discussion gives TransUNet a Dice of 77.48 §4 discussion, while tab. 6 lists 77.49.
• Unclear provenance. The Synapse and ISIC tables do not state whether each row was copied from the source paper or re-run. The reader cannot tell whether the protocols match, for example the split, input size or pre-training. Treat the rows as not directly comparable.
• Few rows, many models. The segmentation discussion counts about 17 reviewed segmentation models §4 discussion, but tab. 6 lists 10 Synapse rows. Some models are only in the taxonomy tables.
• Missing values. TransCASCADE has no FLOPs in tab. 6 (a dash).
• Interpretability is claimed and doubted. §2.3 says Transformers offer better interpretability through attention maps. §10 says attention maps can be vague about which tokens matter, and that propagation methods such as LRP can be numerically unstable on Transformers. Do not treat attention maps as explanations without checking.
• Robustness and pre-training are secondary claims. The review states that ViTs are more robust to adversarial attacks than CNNs §2.3 and that most reviewed methods use ImageNet pre-training §10. Both rest on the cited papers, not on this review's own tests.
• Efficiency numbers are not uniform. Complexity expressions in §10 come from different papers with different definitions of the terms. Compare them only in their original context.
• Search bias. Papers were selected by the authors' judgement of novelty and significance §1. The date range and the completeness of coverage are not stated.

Lineage

• Builds on: the Transformer for sequence modelling §2.1, and the Vision Transformer, cited as arXiv:2010.11929 in the review's bibliography §2.1, §10.
• Compared against: CNN baselines such as U-Net, which the review names as the reference for segmentation designs §2.2, and the CNN-attention hybrids the introduction describes §1.
• Superseded by: not stated.