Semantic Segmentation using Vision Transformers: A survey

In plain words

Image segmentation means labelling every pixel of a picture with what it shows, such as road, sky or building. For years this was done mostly with convolutional networks. The Vision Transformer, which first worked well on whole-image classification, was then adapted to this task. This survey walks through the Transformer models built for segmentation, explains how each one fixes the problems of a plain Transformer on pictures, and lists how well they score on three standard benchmark collections.

It also covers the practical side: the datasets, the loss functions used to train the models, and ways to cope when labelled images are scarce. Its main contribution is the side-by-side table of scores. The authors compile those scores from the original papers and do not run the models themselves.

TL;DR

This is a survey, not a method. It reviews Vision Transformer architectures for semantic segmentation, the datasets and loss functions used with them, and compiles mIoU results on ADE20K, Cityscapes and PASCAL-Context. The table mixes inference protocols and pre-training sets, so rankings across rows are not like-for-like. Use it to find candidate architectures and their reported numbers, then check the source paper.

Problem

Plain Vision Transformers (ViT) produce a single low-resolution feature map, which is poor for dense prediction such as segmentation and detection §1 §2.1. Convolutional networks, by contrast, produce a fixed-size output but capture long-range context poorly §2.1. The survey asks which Transformer designs fix this and how they compare on common benchmarks §1. It states that no earlier survey had compared segmentation models on several benchmarks to identify the best-performing model §1.

Dense prediction also has difficulties the survey lists: intra-class variation, context variation, occlusion and low image resolution §1. Labelled segmentation data is expensive, which the survey treats as a separate problem (data scarcity), addressed in §2.3 by self-supervised pre-training.

Method

This is a survey, so the "method" is its scope and organisation.

1. Background: a ViT splits an image into patches and feeds them as tokens to a Transformer encoder, and self-attention cost grows with the square of the pixel count §2.1 fig. 1.
2. Applications: remote sensing, medical imaging and video segmentation, each with its own challenges and example models §2.2 §2.2.1 §2.2.2 §2.2.3.
3. Data scarcity: the survey describes supervised, transfer and self-supervised learning. It names three self-supervised families (generative, contrastive, and generative-contrastive/adversarial) and notes that segmentation is a common downstream task §2.3 fig. 2.
4. Loss functions: cross-entropy, weighted cross-entropy, focal, Dice and IoU losses, and mean IoU as the evaluation average §2.4.
5. Datasets: PASCAL-Context, ADE20K, KITTI, Cityscapes, IDD, Virtual KITTI and IDDA, summarised in tab. 1 §3.
6. Models: one subsection per architecture (SETR, Swin, Segmenter, SegFormer, PVT v1 and v2, Twins, DPT, HRFormer, Mask2Former), each with its figure §4.1–§4.9, and the benchmark table tab. 2 §4.

The loss functions in §2.4 are the part an implementer can use directly. Written as pseudocode in our own words, from the equations in §2.4:

# Dice loss for one binary mask  [§2.4, Dice eq.]
# Input:  probs (N,) foreground probabilities; target (N,) in {0, 1}
# Output: scalar
def dice_loss(probs, target, eps):
    # overlap term: sum over pixels of g_i * p_i  [§2.4, Dice eq.]
    overlap = sum(target * probs)
    # denominator: sum g_i + sum p_i + eps  [§2.4, Dice eq.]
    denom = sum(target) + sum(probs) + eps
    # loss = 1 - 2 * overlap / denom  [§2.4, Dice eq.]
    return 1 - 2 * overlap / denom

# IoU (Jaccard) loss for one binary mask  [§2.4, IoU eq.]
# Input:  probs (N,); target (N,) in {0, 1}
# Output: scalar
def iou_loss(probs, target, eps):
    # intersection: sum g_i * p_i  [§2.4, IoU eq.]
    intersection = sum(target * probs)
    # union: sum g_i + sum p_i - intersection + eps  [§2.4, IoU eq.]
    union = sum(target) + sum(probs) - intersection + eps
    # loss = 1 - intersection / union  [§2.4, IoU eq.]
    return 1 - intersection / union

# Mean IoU over classes  [§2.4]
# Input:  per_class_iou (C,) one IoU per class
# Output: scalar
def mean_iou(per_class_iou):
    # average of the per-class IoU values  [§2.4]
    return sum(per_class_iou) / len(per_class_iou)

Key equations, as written in §2.4:

CE(p, q) = -\sum_{i=1}^{n} p_i \log q_i \quad \text{[§2.4, cross-entropy eq.]}

F(p_t) = -\alpha_t (1 - p_t)^{\gamma} \log p_t \quad \text{[§2.4, focal eq.]}

D(g, p) = 1 - \frac{2 \sum_i g_i p_i}{\sum_i g_i + \sum_i p_i + \epsilon} \quad \text{[§2.4, Dice eq.]}

IoU(g, p) = 1 - \frac{\sum_i g_i p_i}{\sum_i g_i + \sum_i p_i - \sum_i g_i p_i + \epsilon} \quad \text{[§2.4, IoU eq.]}

Hyperparameters

The survey trains no models of its own, so it states no training hyperparameters. The benchmark table gives only the settings noted per row tab. 2.

Results

Author-reported, compiled by the survey from the original papers. The survey does not state how each number was obtained, and the source setups vary (see Pitfalls). Setup: mIoU (%) on ADE20K, Cityscapes and PASCAL-Context, with the SS / MS pair meaning single-scale and multi-scale inference tab. 2.

The dataset summary from tab. 1:

The full benchmark table tab. 2. Bold marks the best per model family and dataset, as in the source; the source's grey shading (the overall best per dataset) is shown in bold here. Footnotes: ρ patch size and iterations; ℵ single-scale only; § multi-scale only; † ImageNet-1K pre-training; ‡ ImageNet-21K pre-training; * 320k iterations with multi-scale flip testing.

Reading the table: the introduction says the comparison should reveal the best patch size and iteration count per variant §1. The table varies these only for SETR and OCRNet-based HRFormer rows, and the survey does not analyse them.

Paper vs code

No official code. The paper is a survey and has no implementation to release. Checks: the LaTeX source contains no URL or repository link (the only URL is a template comment); no project page is linked; a GitHub repository search for "vision transformer semantic segmentation survey" returned zero results.

Pitfalls

• Mixed protocols in the table. Rows differ in single-scale versus multi-scale inference (ℵ, §, SS / MS), in training length (for example PVT-Large* uses 320k iterations and flip testing tab. 2), and in pre-training data (ImageNet-1K † versus ImageNet-21K ‡). A row-to-row ranking is therefore not a like-for-like comparison. The survey does not normalise these.
• Source of the numbers is not stated. The survey does not say whether each value was taken from the original paper or re-measured. Check the source paper before quoting a number.
• Single-scale and multi-scale numbers come as pairs. Where the table shows two values, the first is single-scale and the second multi-scale tab. 2. Some rows have only one of them (see the footnote symbols).
• Scores are from 2023. The card covers arXiv v1 (2023-05-05). Later models and benchmark results are not in it.
• Loss constants are not given. The focal α and γ and the Dice/IoU ε are written as symbols only §2.4. Pick values from the source papers, not from this survey.
• Cross-entropy averaging is implied, not written. The text says the loss is averaged over all pixels §2.4, but the equation shows a sum over classes for one pixel. The per-pixel averaging step is left to the implementer.
• Unsupported comparative claims. The text says the Dice loss gives higher accuracy than cross-entropy, citing no result §2.4. It says SegFormer-B5 is "4× faster than SETR" §4.4, but tab. 2 has no runtime column and the survey gives no runtime figure.
• Citation mismatches to check. A statement that Transformers are easy to train on large data cites a 1998 vanishing-gradient paper §2.1. A statement that plain ViTs lack segmentation heads cites the Efficient Transformer for remote sensing §2.2.1. Verify both against the original papers before using them.
• Dataset class counts are as stated. Cityscapes is listed with 30 classes and IDD with 34 tab. 1. The survey does not say which label set these counts refer to.
• Figures are third-party. Figures 1–13 are adapted from the original model papers fig. 1–fig. 13. They are not reproduced here; describe them from the source papers instead.

Lineage

• Builds on: the survey cites earlier ViT and segmentation surveys (khan2022transformers, liu2023survey, han2022survey) as the work it extends §1. Their arXiv ids are not given in the paper.
• Compared against: the same three surveys, which the paper says did not compare segmentation models across several benchmarks §1.
• Superseded by: not known from content in this wiki.