---
kind: concept
slug: semantic-segmentation
title: Semantic segmentation
summary: >-
  Assign a class label to every pixel. Covers ViT encoders with light decoders, label-free,
  semi-supervised, domain-adaptive, continual, weakly and text-supervised, few-shot, in-context, promptable,
  open-vocabulary and diffusion-based variants, and domain-specific models (3D medical volumes,
  polyps, aerial and satellite imagery, driving, SLAM), with the trade-offs each one reports.
tags: [computer-vision, semantic-segmentation, vision-transformer, dense-prediction, semi-supervised-learning, domain-adaptation, continual-learning, weak-supervision, unsupervised-learning, diffusion-models, medical-imaging, materials-science, remote-sensing, autonomous-driving, few-shot-learning, in-context-learning, promptable-segmentation, self-supervised-learning, anomaly-detection, gaussian-splatting, slam, synthetic-data, attention-gates, road-extraction, infrared-imaging, simulation-dataset, shape-prior, text-supervision, pretrained-encoder, point-cloud, lidar, multi-task-learning, open-vocabulary, benchmarking, occupancy-prediction, surgical-robotics, long-tail-recognition, prototype-learning, scene-completion, agriculture, survey]
papers: ["1612.05360", "1705.08302", "1706.06480", "1710.07346", "1711.10684", "1801.05746", "1802.02080", "1803.01207", "1804.03999", "1810.08705", "1903.11816", "1903.12212", "1904.07595", "1905.03198", "1906.05797", "1909.13589", "1910.07655", "2011.05499", "2011.07631", "2011.11390", "2011.12954", "2012.03762", "2012.15840", "2101.06742", "2101.07172", "2101.10861", "2102.04306", "2103.01415", "2103.02907", "2103.15597", "2103.16370", "2104.06403", "2105.05633", "2105.11320", "2106.00487", "2106.09681", "2106.12011", "2106.13797", "2107.06307", "2107.12283", "2108.00882", "2108.11250", "2109.03201", "2110.04994", "2112.00390", "2112.03126", "2112.12143", "2112.14757", "2201.03545", "2202.11094", "2203.00962", "2203.03884", "2203.08414", "2203.15102", "2204.07974", "2204.08610", "2205.02833", "2205.10536", "2205.13213", "2206.01191", "2206.09012", "2207.08518", "2207.10866", "2209.01206", "2211.00611", "2211.05778", "2212.08059", "2301.03505", "2302.12242", "2304.03198", "2304.06131", "2310.15161", "2312.02228", "2312.03203", "2404.09556", "2405.05258", "2407.08083", "2408.10618", "2504.01989", "2512.18850", "2603.27781"]
related: [vision-transformer, object-queries, focal-loss, dice-loss, domain-generalization, feature-whitening, hierarchical-vision-transformer, interactive-segmentation, semi-supervised-learning, self-training, self-supervised-learning, contrastive-learning, domain-adaptation, entropy-minimization, continual-learning, knowledge-distillation, bev-representation, open-vocabulary-segmentation, feature-attribution, convolutional-recurrent-network, segmentation-evaluation, benchmark-validity, transfer-learning, data-augmentation, diffusion-models, generative-prior-inverse-problems, anomaly-detection, few-shot-learning, in-context-learning, promptable-segmentation, cost-aggregation, image-to-image-translation, generative-adversarial-network, gaussian-splatting, simultaneous-localization-and-mapping, iterative-closest-point, continuous-convolution, multi-layer-perceptron, multi-task-learning, object-detection, sim-to-real-transfer, efficient-vision-backbone, multi-scale-feature-fusion, instance-segmentation, attention-mechanism, channel-attention, residual-learning, convolutional-neural-network, robot-simulation, shape-prior-regularization, vision-language-pretraining, zero-shot-learning, denoising-autoencoder, long-tailed-recognition, class-balanced-loss, prototype-learning, semantic-scene-completion]
status: draft
updated_at: 2026-11-01
---

## Overview

Semantic segmentation predicts one class per pixel. Most papers here score it by mean intersection
over union (mIoU) or a close relative such as Dice or IoU; the steel micrograph paper reports pixel
and object accuracy instead [arxiv:1706.06480 tab. 5], and so does the land-cover diffusion paper
[arxiv:2206.09012 tab. 1]. The ViT papers report mIoU on ADE20K,
Pascal Context and Cityscapes [arxiv:2012.15840 §4.1] [arxiv:2105.05633 §4.1], and the
semi-supervised paper on PASCAL VOC 2012 and Cityscapes [arxiv:2203.03884 §4.2]. The long-standing
recipe is a fully convolutional encoder that shrinks the image step by step, so that wide context
only becomes available in deep, low-resolution layers, plus add-ons (dilated convolutions, pyramid
pooling, attention modules on top of the backbone) to widen that context
[arxiv:2012.15840 §3.1] [arxiv:2105.05633 §2].

The lines of work below attack different problems. The first asks how to get wide context from a
plain transformer, the second how to train with few labels or none, and the rest how to move a
segmenter to a new domain, a new set of classes, or a new kind of label. Diffusion models are a
newer route: they generate the mask itself, or use a generative prior to constrain it. Specialised
domains (3D medical volumes, endoscopy, steel micrographs, map-view driving, satellite time series,
building footprints, aerial road masks, infrared frames, surgical video) reuse the same core idea with domain-specific encoders and output heads. Three
later settings change what the user supplies at test time instead of retraining: a few labelled
support images ([few-shot](/concepts/few-shot-learning)), a support set given as context
([in-context](/concepts/in-context-learning)), or a click ([promptable](/concepts/promptable-segmentation)).

**Plain ViT encoders.** Both ViT papers replace the convolutional encoder with a plain Vision
Transformer and treat segmentation as sequence-to-sequence over image patches
[arxiv:2012.15840 §1] [arxiv:2105.05633 §1]. What they share:

- **Patch tokens at fixed resolution.** The image is split into $P \times P$ patches, each linearly
  projected and given a learned position embedding, giving $N = HW/P^2$ tokens. The token grid is
  never downsampled; every layer runs global self-attention over all $N$ tokens
  [arxiv:2012.15840 §3.2] [arxiv:2105.05633 §3.1].
- **Pre-trained ViT/DeiT weights.** The encoder is initialised from an ImageNet classifier and the
  stored position embeddings are interpolated in 2D to the fine-tuning grid
  [arxiv:2012.15840 §4.1] [arxiv:2105.05633 §4.2]. Without this, results collapse (see When to use
  which).
- **A light decoder** maps the $H/P \times W/P$ feature map to per-class scores and upsamples
  bilinearly to full resolution; training uses per-pixel cross-entropy [arxiv:2012.15840 §3.3]
  [arxiv:2105.05633 §3.2].
- **An MMSegmentation-style recipe**: SGD with weight decay 0, polynomial LR decay, random resize
  0.5 to 2.0, random crop and horizontal flip, sliding-window inference, and multi-scale testing at
  scales 0.5 to 1.75 plus flip [arxiv:2012.15840 §4.1] [arxiv:2105.05633 §4.2].

The ViT variants differ almost entirely in the decoder.

**Semi-supervised training.** U2PL keeps a convolutional network (ResNet-101 with DeepLabv3+) and
attacks the labeling cost instead: it trains on a few labeled images and many unlabeled ones in a
teacher-student setup [arxiv:2203.03884 §3.1] [arxiv:2203.03884 §4.1]. Its question is what to do
with the pixels the teacher is unsure about.

## Variants

### SETR: ViT encoder with convolutional decoders

SETR runs a ViT encoder on 16x16 patches (T-Base: 12 layers, width 768; T-Large: 24 layers, width
1024) and attaches one of three decoders to the 1/16 feature map [arxiv:2012.15840 §3.2]
[arxiv:2012.15840 tab. 1] [arxiv:2012.15840 §3.3]:

- **Naive**: two 1x1 convolutions (with sync BN and ReLU between them) down to the class count, then
  a single 16x bilinear upsample.
- **PUP** (progressive upsampling): four rounds of convolution plus 2x upsampling.
- **MLA** (multi-level aggregation): takes $M$ evenly spaced encoder layers, reduces each with
  convolutions, fuses neighbouring streams top-down by addition, upsamples each stream 4x,
  concatenates and upsamples 4x again.

Auxiliary segmentation heads on intermediate layers are trained alongside the main head
[arxiv:2012.15840 §4.1]. Author-reported, the best decoder depends on the dataset: PUP leads on
Cityscapes (82.15 mIoU val, multi-scale) while MLA leads on ADE20K (50.28 val, multi-scale) and
Pascal Context (55.83) [arxiv:2012.15840 tab. 4] [arxiv:2012.15840 tab. 5]
[arxiv:2012.15840 tab. 6]. The large models carry 305 to 318M parameters, against 68.59M for the
ResNet-101 FCN baseline [arxiv:2012.15840 tab. 2]. Card: [Rethinking Semantic Segmentation from a
Sequence-to-Sequence Perspective with Transformers](/papers/2012.15840).

The same paper also tests a **hybrid** that feeds a ResNet-50 feature map into the transformer plus
Naive decoder; with DeiT weights it reaches 78.28 on Cityscapes val at 80k iterations, below
SETR-PUP-DeiT's 79.45 [arxiv:2012.15840 §4.1] [arxiv:2012.15840 tab. 2].

### Segmenter: ViT encoder with a linear or mask-transformer decoder

Segmenter keeps the decoder transformer-only [arxiv:2105.05633 §3]:

- **Linear**: one point-wise linear layer from token width to $K$ classes per patch, then bilinear
  upsampling [arxiv:2105.05633 §3.2].
- **Mask transformer**: $K$ learnable class embeddings, one per class, are appended to the patch
  tokens and processed by $M$ extra transformer layers ($M = 2$); each class mask is the scalar
  product of the L2-normalised patch outputs with that class's output embedding
  [arxiv:2105.05633 §3.2] [arxiv:2105.05633 eq. 4] [arxiv:2105.05633 §4.3]. This is the
  one-slot-per-class form of [object queries](/concepts/object-queries).

The mask decoder adds 0.4 to 1.6 mIoU over the linear one on ADE20K val, single-scale (for example
Seg-B†/16 47.10 to 48.70, Seg-L/16 50.71 to 51.30), though not in every object-size bucket:
Seg-B-Mask/16 scores lower than linear Seg-B/16 on large objects (52.66 vs 53.28)
[arxiv:2105.05633 tab. 4]. The paper explores encoder size from Tiny
(6M) to Large (307M) and patch sizes 8, 16 and 32 [arxiv:2105.05633 tab. 1]
[arxiv:2105.05633 tab. 3]. Best author-reported results come from Seg-L-Mask/16: 53.63 mIoU on
ADE20K val (multi-scale, trained at 640 crops), 59.0 on Pascal Context and 81.3 on Cityscapes val
[arxiv:2105.05633 tab. 6] [arxiv:2105.05633 tab. 7] [arxiv:2105.05633 tab. 8]
[arxiv:2105.05633 §4.2]. Card: [Segmenter: Transformer for Semantic
Segmentation](/papers/2105.05633).

### Hybrid CNN stem with a ViT encoder (TransUNet)

TransUNet keeps a ResNet-50 stem. Each cell of its 14x14 feature map becomes one token for a
12-layer ViT-B encoder, so no 16x16 pixel patch is cut from the image [arxiv:2102.04306 §3.2]
[arxiv:2102.04306 §4.2]. A cascaded upsampler decodes the tokens back to full resolution and
receives skip connections from the CNN at 1/8, 1/4 and 1/2 of the input; three skips is the best
setting the paper tests [arxiv:2102.04306 §3.2] [arxiv:2102.04306 §4.4]. On Synapse multi-organ
CT (224 input, 18 training and 12 evaluation cases) it reports 77.48 average DSC, against 75.57 for
R50-AttnUNet and 74.68 for R50-U-Net [arxiv:2102.04306 tab. 1]. The hybrid stem matters: with the
same upsampler, R50-ViT reaches 71.29 against 67.86 for a pure ViT with the same decoder [arxiv:2102.04306 tab. 1]. On
ACDC cardiac MRI it reports 89.71 average DSC, against 87.55 for R50-U-Net [arxiv:2102.04306 tab. 5].
A 512 input raises Synapse to 84.36 DSC, but the sequence is about five times longer and the paper
calls the cost much larger [arxiv:2102.04306 tab. 2] [arxiv:2102.04306 §4.4]. The paper does not
state its loss, its learning-rate schedule or its checkpoint rule, so reproduce from the card's
notes rather than the text alone. Card: [TransUNet: Transformers Make Strong Encoders for Medical
Image Segmentation](/papers/2102.04306).

### Two pyramids with class-token fusion (HiFormer)

[HiFormer](/papers/2207.08518) feeds a three-level CNN into a three-level Swin pyramid through 1x1
convolutional skips [arxiv:2207.08518 §3.1.1] [arxiv:2207.08518 §3.1.2]. Only the
two outer levels are fused. Each gets a class token that attends to the other level's patch tokens,
and the cost is linear in the token count because the class token is the only query
[arxiv:2207.08518 §3.1.3] [arxiv:2207.08518 eq. 3]. The decoder then sums the two outer levels and
upsamples to full resolution [arxiv:2207.08518 §3.2].

Author-reported on Synapse (8 abdominal organs), HiFormer-B reaches average Dice 80.39 and HD 14.70,
against 79.13 and 21.55 for Swin-Unet and 77.48 and 31.69 for TransUnet [arxiv:2207.08518 tab. 2].
Removing the fusion module drops Dice to 77.15 and raises HD to 16.88 [arxiv:2207.08518 tab. 6]. The
paper reports +3.24 Dice from this module [arxiv:2207.08518 §5]. The variants differ in CNN depth and
fusion size: B uses a ResNet-50 and 25.51 M parameters, against 23.25 M for S and 29.52 M for L
[arxiv:2207.08518 tab. 1] [arxiv:2207.08518 tab. 5]. Two points need care before implementing. The
paper does not say how the class-token output reaches the decoder, which reads only patch tokens
[arxiv:2207.08518 eq. 3] [arxiv:2207.08518 §3.2]. The released forward pass also does not pass the
fused output to the decoder, so it does not reproduce the Synapse gain as written (see its card).
Card: [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image
Segmentation](/papers/2207.08518). See also [hierarchical vision transformer](/concepts/hierarchical-vision-transformer).

### Encoder scale and patch size

Within one decoder, the encoder dominates. On ADE20K val (linear decoder, single-scale), Segmenter
goes from 39.03 (Ti/16) to 45.37 (S/16), 48.06 (B/16) and 50.71 (L/16); SETR likewise reports T-Large
above T-Base [arxiv:2105.05633 tab. 3] [arxiv:2012.15840 §4.2]. Smaller patches help too: Seg-B/32
43.07, Seg-B/16 48.06, Seg-B/8 49.54, but throughput falls from 516 to 92 to 7 images/s on one V100
[arxiv:2105.05633 tab. 3]. Since $N = HW/P^2$, halving $P$ quadruples the token count
[arxiv:2105.05633 §3.1].

### Pyramid transformer backbone with a Semantic FPN head (PVT v2)

PVT v2 is a four-stage pyramid transformer, so its features come at several resolutions and a
decoder can fuse them [arxiv:2106.13797 §3.5]. Its attention pools keys and values to a fixed 7x7
grid, which makes the attention cost linear in the number of image tokens [arxiv:2106.13797 §3.2]
[arxiv:2106.13797 eq. 2]. Queries still come from every token, so the projections still scale with
the input [arxiv:2106.13797 §3.2]. Overlapping patch embedding and a 3x3 depthwise convolution in
the feed-forward block are its other changes [arxiv:2106.13797 §3.3] [arxiv:2106.13797 §3.4].

For ADE20K the segmentation head is Semantic FPN, trained with AdamW at learning rate 1e-4 for 40k
iterations at batch 16, with polynomial decay and 512 x 512 crops [arxiv:2106.13797 §4.3].
Author-reported mIoU is 45.2 for PVT v2-B2, at 29.1 M parameters and 45.8 GFLOPs at 512 x 512,
against 39.8 for PVT v1-Small (28.2 M, 44.5 GFLOPs) and 36.7 for ResNet-50 (28.5 M, 45.6 GFLOPs)
[arxiv:2106.13797 tab. 5]. The linear-attention variant B2-Li reaches 45.1 at 26.3 M and 41.0
GFLOPs, and B5 reaches 48.7 [arxiv:2106.13797 tab. 5]. The repository's segmentation configs cover
PVT v1 only [code: segmentation/configs/sem_fpn/PVT/fpn_pvt_s_ade20k_40k.py#L20], so the v2
segmentation setup has to be built from the paper's text. Card: [PVT v2: Improved Baselines with
Pyramid Vision Transformer](/papers/2106.13797). See also [vision transformer](/concepts/vision-transformer).

### Pyramid pooling attention for dense features (P2T)

P2T builds attention keys and values from a pyramid of average-pooled feature maps at four scales.
Each pooled map passes through a shared 3x3 depthwise convolution, and the results are concatenated
into a fixed number of tokens [arxiv:2106.12011 §III-B.1] [arxiv:2106.12011 eq. 2-4]. Queries still
come from the full input, and the feed-forward block is an inverted bottleneck with a depthwise
convolution [arxiv:2106.12011 §III-B.2] [arxiv:2106.12011 eq. 10]. With Semantic FPN on ADE20K,
author-reported mIoU is 46.7 for P2T-Small and 48.7 for P2T-Base [arxiv:2106.12011 tab. 3].
Segmentation training uses AdamW at learning rate 1e-4 for 80k iterations at batch 16 with 512 x 512
crops [arxiv:2106.12011 §IV-B]. Fixed pooled sizes save 2.9 to 3.3 GB of memory [arxiv:2106.12011
tab. 8]. The pooling rounding and the head count are not stated [arxiv:2106.12011 §III-B.1]. Card:
[P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011). See also
[vision transformer](/concepts/vision-transformer).

### Channel-covariance attention backbone (XCiT)

XCiT replaces token-to-token attention with attention across feature channels. Queries and keys are
l2-normalised over the token axis, and a channel-by-channel cross-covariance matrix is softmaxed
[arxiv:2106.09681 §3.2] [arxiv:2106.09681 eq. 1]. The cost grows linearly with the number of tokens,
and every layer keeps the same token count [arxiv:2106.09681 §3.3]. A depthwise convolution after the
attention adds local patch interaction before the feed-forward block [arxiv:2106.09681 fig. 1].
Author-reported ADE20K mIoU (single scale) is 44.2 for XCiT-S12/8 with a Semantic FPN head and 48.1
for XCiT-S24/8 with UperNet [arxiv:2106.09681 tab. 6]. Segmentation training uses AdamW at learning
rate 6e-5 and batch 16, for 80k iterations with FPN and 160k with UperNet [arxiv:2106.09681 §4.3].
The softmax orientation in eq. 1 is ambiguous and differs from the released code, which the card
recommends [arxiv:2106.09681 app. C]. Without the l2 normalisation, S12/16 fails [arxiv:2106.09681
tab. 4]. Card: [XCiT: Cross-Covariance Image Transformers](/papers/2106.09681).

### High-resolution backbone with cheap channel weighting (Lite-HRNet)

Lite-HRNet keeps the parallel high-resolution branches of HRNet but builds them from shuffle blocks.
It replaces the costly 1x1 convolutions with conditional channel weighting: pooled features from
every branch set per-channel weights for the others [arxiv:2104.06403 §3.2] [arxiv:2104.06403 eq. 2]
[arxiv:2104.06403 eq. 4]. A Lite module is two weighting blocks and one fusion block [arxiv:2104.06403
tab. 1]. For segmentation the models are trained from scratch, without ImageNet pretraining, using
SGD with polynomial learning rate 1e-2 for 160k iterations at batch 16 [arxiv:2104.06403 §4.4]. The
Cityscapes validation mIoU is 73.8 for the 18-channel model and 76.0 for the 30 model [arxiv:2104.06403
tab. 8]. The paper does not state the channel-reduction ratios, and the released repository has no
segmentation code, so table 8 cannot be reproduced from it [arxiv:2104.06403 §4.4]. Card:
[Lite-HRNet: A Lightweight High-Resolution Network](/papers/2104.06403).

### Learned upsampling instead of dilated backbones (FastFCN)

FastFCN drops the dilation from a standard backbone and recovers wide context with a learned head,
Joint Pyramid Upsampling (JPU). JPU takes the outputs of the last three stages, maps each to a common
width, upsamples the two deeper maps to the shallowest size, concatenates, and applies four parallel
separable convolutions with dilations 1, 2, 4 and 8 [arxiv:1903.11816 §3.2] [arxiv:1903.11816 §3.3.3].
A context head (PSP, ASPP or Encoding) follows [arxiv:1903.11816 §3.2]. On PASCAL-Context with
ResNet-50, Encoding with JPU reaches 51.05 mIoU, against 49.91 for dilated EncNet [arxiv:1903.11816
tab. 1]. At 512 x 512 on a Titan Xp it runs at 37.56 FPS, against 18.77 for the dilated version
[arxiv:1903.11816 tab. 2]. On ADE20K with ResNet-101, dilated EncNet is slightly higher, 44.65 against
44.34 [arxiv:1903.11816 tab. 4]. The "more than three times" reduction in computation is stated in
the text without a FLOP or memory table [arxiv:1903.11816 §1] [arxiv:1903.11816 §5]. The text gives
52.10 and 54.03 mIoU for the 59-class protocol, while the abstract gives 53.13 for ResNet-101; see
Contradictions. Card: [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic
Segmentation](/papers/1903.11816).

### Pure ConvNet backbone modernised from ResNet (ConvNeXt)

ConvNeXt starts from a ResNet-50 and applies a sequence of changes borrowed from transformer training
and design, ending in a block of a 7x7 depthwise convolution, LayerNorm, a 4x-wide inverted bottleneck
with one GELU, and layer scale [arxiv:2201.03545 §2.4] [arxiv:2201.03545 §2.5] [arxiv:2201.03545
§2.6] [arxiv:2201.03545 fig. 4]. The stem is a 4x4 stride-4 patchify convolution, and downsampling uses
separate 2x2 stride-2 convolutions with LayerNorm [arxiv:2201.03545 §2.2]. With UperNet on ADE20K,
multi-scale mIoU is 46.7 for ConvNeXt-T against 45.8 for Swin-T [arxiv:2201.03545 tab. 4].
Pre-training uses AdamW at learning rate 4e-3, batch 4096 and weight decay 0.05 [arxiv:2201.03545
tab. 5]. Depthwise convolutions are slower than dense ones at equal FLOPs [arxiv:2201.03545 §4]. The
kernel size is inconsistent inside the paper: §2.5 says 7x7 and appendix C says 5x5 [arxiv:2201.03545
app. C]. Card: [A ConvNet for the 2020s](/papers/2201.03545). See also
[convolutional neural network](/concepts/convolutional-neural-network).

### Local-global attention with pooled heads (HiLo, LITv2)

HiLo splits the heads of each attention layer into two groups. A fraction alpha of the heads (Lo-Fi)
attends from full-resolution queries to keys and values average-pooled by an s x s window, and the
rest (Hi-Fi) run self-attention inside non-overlapping s x s windows [arxiv:2205.13213 §4.1]
[arxiv:2205.13213 eq. 6]. LITv2 uses HiLo in its last two stages [arxiv:2205.13213 §4.2]. With a
Semantic FPN head on ADE20K, author-reported mIoU is 47.2 for LITv2-B [arxiv:2205.13213 tab. 3].
Classification training is 300 epochs on eight V100 GPUs at total batch 1,024 [arxiv:2205.13213 §5.1].
Speed rankings change across A100, V100 and RTX 6000 GPUs [arxiv:2205.13213 tab. 6]. For 12 heads,
alpha times the head count is not an integer, and the released code rounds it down [arxiv:2205.13213
§4.1]. Card: [Fast Vision Transformers with HiLo Attention](/papers/2205.13213).

### Latency-driven mobile backbones (EfficientFormer and EfficientFormerV2)

EfficientFormer is a pure transformer with convolution-style early blocks: two stride-2 3x3
convolutions as the stem, then pooling-mixer blocks without layer norm, and attention only in the last
stage [arxiv:2206.01191 eq. 3] [arxiv:2206.01191 eq. 4] [arxiv:2206.01191 §4.1]. Its depth and width
come from a slimming step that uses a per-block latency table measured on an iPhone 12, applied to a
supernet trained with Gumbel-softmax sampling [arxiv:2206.01191 §4.2] [arxiv:2206.01191 alg. 1].
Author-reported ImageNet-1K top-1 is 83.3 at 7.0 ms for the largest variant, and ADE20K mIoU is 45.1
for it [arxiv:2206.01191 tab. 1] [arxiv:2206.01191 tab. 2]. Latency is per image through Core ML,
averaged over 1000 runs, and the batch size is not stated [arxiv:2206.01191 tab. 1]. The learning rate
is 1e-3 scaled by batch size over 1024 [arxiv:2206.01191 §5.1].

EfficientFormerV2 keeps the family but adds a depthwise convolution in every feed-forward block,
attention only in the last two stages, and a joint size-speed search [arxiv:2212.08059 §3.1]
[arxiv:2212.08059 §4.2.1]. The search score weights latency and size, and each search evaluation takes
about four GPU-minutes [arxiv:2212.08059 §4.2.3]. Author-reported ADE20K mIoU is 45.2 for the L model
and 42.4 for S2 [arxiv:2212.08059 tab. 3]; ImageNet-1K top-1 is 83.3 at 2.7 ms for L
[arxiv:2212.08059 tab. 2]. The paper states the learning rate per 1024 batch and 16 GPUs, but not
the per-GPU batch size [arxiv:2212.08059 §5.1]. Cards: [EfficientFormer: Vision Transformers at
MobileNet Speed](/papers/2206.01191) and [Rethinking Vision Transformers for MobileNet Size and
Speed](/papers/2212.08059).

### Deformable-convolution CNN backbones (InternImage)

InternImage builds on DCNv3, a 3x3 deformable convolution that starts from DCNv2 and makes three
changes. One value projection is shared across the sampling points, the channels are split into groups
that each have their own offsets, and the weights over the sampling points of each group are normalised
with a softmax [arxiv:2211.05778 §3.1] [arxiv:2211.05778 eq. 2]. Blocks look like transformer blocks,
with layer norm, a feed-forward network and residual connections [arxiv:2211.05778 §3.2]
[arxiv:2211.05778 fig. 3]. Author-reported ImageNet-1K top-1 is 83.5 for the T model, against 82.1 for
ConvNeXt-T [arxiv:2211.05778 tab. 2]. With Mask2Former on ADE20K, the H model reaches 62.9 mIoU
[arxiv:2211.05778 tab. 5]. The largest models are pre-trained on ImageNet-22K, and H uses a joint
pre-training set and M3I [arxiv:2211.05778 §4.1]. The joint set is not listed in the paper
[arxiv:2211.05778 app. A]. Sharing the projection saves 42.0% of the parameters and 84.2% of GPU memory
at the H scale [arxiv:2211.05778 §4.3]. Card: [InternImage: Exploring Large-Scale Vision Foundation
Models with Deformable Convolutions](/papers/2211.05778).

### Mamba and state-space hybrid backbones (MambaVision)

MambaVision starts with two convolutional stages. In its last two stages, each block's layers are split
in half: the first half use a Mamba mixer, which runs a selective scan on half of the channels and a
convolutional gate on the other half, and the second half use self-attention [arxiv:2407.08083 §3.2.2]
[arxiv:2407.08083 eq. 7]. With UperNet at 512 x 512 on ADE20K, author-reported mIoU is 46.0 for Tiny and
49.1 for Base [arxiv:2407.08083 tab. 3]. ImageNet-1K top-1 is 82.3 for Tiny, at 6,298 images per second
on an A100 at batch 128 [arxiv:2407.08083 tab. 1]. The paper's stage widths in its appendix table are
twice those in the released code [arxiv:2407.08083 tab. S.2]. The scan kernel comes from an external
package that the repository does not include. The text says MambaVision-B has 56% fewer GFLOPs than
MaxViT-B, but table 1 gives about 36% [arxiv:2407.08083 §5.1]. The card records a non-commercial code
licence and non-commercial weights, so reuse is not commercial-safe. Card: [MambaVision: A Hybrid
Mamba-Transformer Vision Backbone](/papers/2407.08083).

### Receptive-field attention convolutions (RFAConv)

RFAConv is a drop-in replacement for a k x k convolution. Attention maps are computed per window
instead of being shared by overlapping windows, and each window's k² positions get their own softmax
weights [arxiv:2304.03198 §3.1] [arxiv:2304.03198 eq. 2]. The window values are expanded, weighted,
reshaped into k x k blocks and passed through a stride-k convolution [arxiv:2304.03198 §3.3]
[arxiv:2304.03198 eq. 3]. RFCAConv builds on coordinate attention in the same way
[arxiv:2304.03198 §3.3]. Author-reported ImageNet-1k top-1 for ResNet-18 rises from 69.59 to 71.23 with
RFAConv [arxiv:2304.03198 tab. 3]. On PASCAL VOC 2012 with DeepLabV3+ at stride 16, mIoU is 64.6
for the baseline and 65.4 with RFAConv, and 68.0 with RFCAConv [arxiv:2304.03198 tab. 10]. The expanded
tensor is k² times the size of the input, which the authors say limited-storage devices may not afford
[arxiv:2304.03198 §4.6]. The paper describes the Unfold step as parameterless, while the released code
adds a learned 1x1 grouped convolution and batch norm. Gains are uneven; coordinate attention does
better on segmentation [arxiv:2304.03198 §4.6]. Card: [RFAConv: Receptive-Field Attention Convolution
for Improving Convolutional Neural Networks](/papers/2304.03198). See also
[channel attention](/concepts/channel-attention).

### Residual encoder-decoders with summation skips (FusionNet)

FusionNet is a U-Net-style network for neuron boundaries in electron-microscopy stacks. Inside every
level it places residual blocks, and on the long skip connections it adds the encoder map to the
decoder map instead of concatenating them [arxiv:1612.05360 §3.1] [arxiv:1612.05360 tab. 1]. It
takes 640x640 tiles, uses eight-fold orientation enrichment with elastic warps and noise, and
averages the eight predictions at test time [arxiv:1612.05360 §3.2]. Author-reported, it beats the
original U-Net on the Drosophila test set (V_rand 0.97804 vs 0.97276) and on a zebrafish test
volume (Dice 0.9461 vs 0.9085) [arxiv:1612.05360 tab. 2] [arxiv:1612.05360 tab. 3]. The optimiser,
learning rate, batch size and epoch count are not stated, so the training recipe must be chosen by
the implementer [arxiv:1612.05360 §3.3]. Card: [FusionNet: A deep fully residual convolutional
neural network for image segmentation in connectomics](/papers/1612.05360).

### Attention gates on skip connections (Attention U-Net)

Attention U-Net adds an additive attention gate to each skip connection of a 3D U-Net. The gate
combines the skip features with a coarser gating signal and produces one sigmoid coefficient per
voxel, or one per class; the skip features are then scaled by it [arxiv:1804.03999 §2]
[arxiv:1804.03999 eq. 1] [arxiv:1804.03999 eq. 2]. The lowest skip connections are not gated, and
each decoder scale gets an auxiliary output [arxiv:1804.03999 §2]. The stated aim is to replace the
two-stage cascade, which first localises a small organ and then segments it, with one network
[arxiv:1804.03999 §1].

Author-reported on CT-150 with 120 training and 30 test scans, pancreas DSC is 0.840 for Attention
U-Net against 0.814 for U-Net; the ± values are shown as printed, and the paper does not say what
they measure [arxiv:1804.03999 tab. 1] [arxiv:1804.03999 §3]. On the public Pancreas-CT set the
gain is smaller before fine-tuning, 0.712 against 0.690 [arxiv:1804.03999 tab. 3]. The gates cost
6.40 M parameters against 5.88 M, and 0.179 s against 0.167 s per forward pass [arxiv:1804.03999
tab. 1]. The significance tests behind the p-values are not named [arxiv:1804.03999 §3]. The
released code differs from the paper in its optimiser and in some layers, so check its card before
reproducing [arxiv:1804.03999 §3]. Card: [Attention U-Net: Learning Where to Look for the
Pancreas](/papers/1804.03999). See also [attention mechanism](/concepts/attention-mechanism).

### Position-aware channel attention in a DeepLabV3 model (coordinate attention)

Coordinate attention is a channel-attention block for mobile networks that keeps position. It
pools along height and width separately and produces one gate per row and one per column
[arxiv:2103.02907 §3.2.1] [arxiv:2103.02907 §3.2.2]. The segmentation test adds it to a DeepLabV3
model with a MobileNetV2 backbone; the ASPP branches use depthwise-separable 3x3 convolutions and
weight decay is 4e-5 [arxiv:2103.02907 §4.4.2]. See also [channel attention](/concepts/channel-attention).

Author-reported mIoU on Pascal VOC 2012 val is 73.32 at output stride 16 and 73.96 at output stride
8 with coordinate attention, against 70.84 and 71.82 for the plain backbone and 71.69 and 72.52 with
SE [arxiv:2103.02907 tab. 8]. CBAM is 0.4 above the baseline at stride 16 and 0.15 below it at
stride 8 [arxiv:2103.02907 tab. 8]. On Cityscapes val at output stride 8 the paper reports 74.0,
against 71.4 for the baseline, 72.2 with SE and 71.4 with CBAM [arxiv:2103.02907 tab. 9]. The block
adds 0.06 M parameters over SE on ImageNet, with reduction ratios that differ from SE's
[arxiv:2103.02907 tab. 1]. The repository holds the block and a MobileNetV2 model but no
segmentation code [code: README.md#L7] [code: README.md#L56] [code: README.md#L82], so these
numbers come from the paper's tables and were not re-run. Card: [Coordinate Attention for Efficient
Mobile Network Design](/papers/2103.02907).

### Training-only shape prior on the output (ACNN)

ACNN adds a shape term to the segmentation loss and drops it at inference. A convolutional
autoencoder, trained on label maps, maps the predicted and the true label maps to 64-dimensional
codes, and the loss penalises the squared distance between the two codes [arxiv:1705.08302 §2.2]
[arxiv:1705.08302 eq. 1]. For cardiac MR super-resolution, where the output is an intensity image,
a separate predictor maps images into the same code space and is used only during training
[arxiv:1705.08302 §2.4].

Author-reported on cardiac MR, the endocardial Dice is 0.939 against 0.923 for a 3D network with
sub-pixel up-sampling, and the myocardial Dice is 0.811 against 0.773 [arxiv:1705.08302 tab. 1].
The Hausdorff differences are not significant in that table [arxiv:1705.08302 tab. 1]. On CETUS 3D
ultrasound, the ejection-fraction correlation is 0.913 against 0.885 for an FCN [arxiv:1705.08302
tab. 2]. For super-resolution, SSIM is 0.796 against 0.784 for a 3D U-Net, at 0.06 s per
120 x 120 x 12 input [arxiv:1705.08302 tab. 3]. The autoencoder's training schedule, the input
the encoder receives and the loss reductions are not stated, so the shape term cannot be
reproduced from the text alone [arxiv:1705.08302 §2.2] [arxiv:1705.08302 §3.2]. Card: [ACNN](/papers/1705.08302). See also [shape-prior regularization](/concepts/shape-prior-regularization).

### Residual U-Net for aerial road masks (Deep ResUnet)

Deep ResUnet replaces the plain convolution blocks of U-Net with full pre-activation residual units
and drops the cropping before each concatenation, because input and output have the same size
[arxiv:1711.10684 §2.1.3] [arxiv:1711.10684 fig. 2]. The text gives 15 convolution layers against
23 for U-Net, and about 7.8 M parameters against 30.6 M [arxiv:1711.10684 §2.1.3]
[arxiv:1711.10684 §3.4]. Training uses an MSE loss with SGD on 30,000 random 224 x 224 patches,
without augmentation; momentum and weight decay are not stated [arxiv:1711.10684 §3.2]
[arxiv:1711.10684 §2.2].

The metric is a relaxed break-even point with a 3-pixel slack, not IoU or mIoU. On the 49-image
Massachusetts roads test set it reports 0.9187, against 0.9053 for U-Net and 0.9047 for Saito et
al. [arxiv:1711.10684 tab. II]. The gap to U-Net is 0.013 points, and the paper reports no seeds or
spread, so the difference is not established [arxiv:1711.10684 tab. II]. Only one dataset is
tested [arxiv:1711.10684 §3.1]. The paper has no official code. Card: [Road Extraction by Deep
Residual U-Net](/papers/1711.10684).

### ImageNet-pretrained VGG encoder in a U-Net (TernausNet)

TernausNet replaces the encoder of a U-Net with the convolutional part of VGG11, initialised from
ImageNet weights, and segments buildings against background in aerial images [arxiv:1801.05746 §2]
[arxiv:1801.05746 fig. 1]. Training minimises binary cross-entropy minus the log of a soft Jaccard
term, and the probability map is thresholded at 0.3 [arxiv:1801.05746 eq. 4] [arxiv:1801.05746 §2].
Adam with learning rate 0.001 runs for 100 epochs on random 768 x 768 crops [arxiv:1801.05746 §3].

Author-reported validation IoU on the Inria aerial set is 0.593 with random (LeCun-uniform)
initialisation, 0.686 with the ImageNet encoder, and 0.687 with a full network pretrained on Carvana
[arxiv:1801.05746 §3]. These come from 30 validation images and one run
per setting, with no spread reported, so the gain is one experiment rather than a broad result
[arxiv:1801.05746 §3]. The text describes seven encoder convolutions, while the released code uses
eight, and the skip connections and bottleneck differ in detail (see its card)
[arxiv:1801.05746 §2]. Card: [TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image
Segmentation](/papers/1801.05746). See also [transfer learning](/concepts/transfer-learning).

### Surgical instruments: U-Net-family encoders for tool masks

Three U-Net-style networks and one LinkNet segment robot instruments in surgical video frames,
at three levels: a binary tool mask, three tool parts, and seven instrument types
[arxiv:1803.01207 §2.1] [arxiv:1803.01207 §3]. U-Net is trained from scratch, while TernausNet-11
and TernausNet-16 use VGG encoders and LinkNet-34 uses a ResNet-34 encoder [arxiv:1803.01207 §2.2].
The loss is cross-entropy plus a soft Jaccard term, written H − log J [arxiv:1803.01207 §2.3]
[arxiv:1803.01207 eq. 3]. The paper applies a 0.3 threshold to the binary output [arxiv:1803.01207 §2.3].

Author-reported on the MICCAI 2017 challenge training data [arxiv:1803.01207 §2.1]: binary IoU is
83.60 for TernausNet-16 against 75.44 for U-Net; parts IoU is 65.50 for TernausNet-16 against 48.41
for U-Net; instrument-type IoU is 34.61 for TernausNet-11 [arxiv:1803.01207 tab. 1]. LinkNet-34 is
the fastest on the binary task, at 88 ms on a GTX 1080 Ti against 184 ms for TernausNet-16, but its
parts IoU is 34.55 [arxiv:1803.01207 tab. 1]. The paper attributes the weak multi-class results to
few training examples for some instrument types [arxiv:1803.01207 §3]. It does not say which split
its table uses, and the released evaluation differs from the paper (see its card), so treat the
numbers as one study's results. Card: [Automatic Instrument Segmentation in Robot-Assisted Surgery
Using Deep Learning](/papers/1803.01207). See also [transfer learning](/concepts/transfer-learning).

### Pixel-wise FCN with per-object voting for microscopy

MVFCNN labels steel micrographs with a fully convolutional network. The backbone is VGG16 with its
fully connected layers turned into convolutions, plus up-sampling and skip layers [arxiv:1706.06480
§4.2] [arxiv:1706.06480 §3.1]. Large SEM images are cut into 1000x1000 px windows, every pixel is
labelled, the labels are stitched back together, and each object takes the majority label of its
pixels [arxiv:1706.06480 §4.2]. Objects are never resized, which the paper argues keeps the texture
a classifier needs [arxiv:1706.06480 §4.1] [arxiv:1706.06480 §6].

Training runs in three stages, FCN-32s, then FCN-16s, then FCN-8s, each starting from the weights
of the previous one. The learning rates are 1e-10, 1e-11 and 3e-12, and the paper says larger rates
made the loss explode [arxiv:1706.06480 §4.3.3]. Per-class sliding-window strides balance the patch
counts across classes [arxiv:1706.06480 §4.3.4]. The exact per-class strides are not stated.

Author-reported on 21 SEM and LOM image pairs, with one split of 11 training and 10 test images
[arxiv:1706.06480 §5.1]: 93.94% object accuracy on SEM, against 66.50% for a fine-tuned VGG16 on
cropped objects and 48.89% for the earlier Pauly et al. method [arxiv:1706.06480 tab. 1]. Training
from scratch gives 55.50%, fine-tuning 87.98%, class balancing 90.97% and augmentation 93.94%
[arxiv:1706.06480 tab. 2]. On LOM images the same method reaches 70.14% [arxiv:1706.06480 tab. 3].
Bainite is the weakest class, with 37.32% pixel accuracy in the best setup [arxiv:1706.06480 tab. 6].
Card: [Advanced Steel Microstructural Classification by Deep Learning Methods](/papers/1706.06480).

### Map-view and 3D segmenters for driving and volumes

Four papers in this group move the segmenter's output space rather than its encoder.

- **Vehicle and road maps from six cameras.** The Cross-view Transformer encodes each camera with
  EfficientNet-B4, uses cross-view attention to lift the features into a 25x25 map embedding, and
  decodes it to 200x200 [arxiv:2205.02833 §3.2] [arxiv:2205.02833 §4]. Each camera key carries a
  positional embedding built from its intrinsics and extrinsics, so no explicit depth or homography
  is computed [arxiv:2205.02833 §3.1]. On nuScenes Setting 2 it reaches vehicle IoU 36.0 with 5M
  parameters and 35 FPS, against 35.8 with 7M parameters and 8 FPS for static FIERY
  [arxiv:2205.02833 tab. 1]. The paper reports 37.5 against FIERY's 37.7 in Setting 1
  [arxiv:2205.02833 tab. 1]. Card: [Cross-view Transformers for real-time Map-view Semantic
  Segmentation](/papers/2205.02833). See also [BEV representation](/concepts/bev-representation).
- **Vector HD-map elements.** HDMapNet predicts lane dividers, lane boundaries and pedestrian
  crossings in bird's-eye view. Camera features go through a learned per-camera MLP and are warped
  with extrinsics; LiDAR uses a PointPillars-style encoder; three heads output semantics, instance
  embeddings and lane direction [arxiv:2107.06307 §3.1]. Camera and LiDAR fusion reaches 44.5 IoU
  and 30.6 mAP, against 32.9 IoU and 22.7 mAP for cameras alone [arxiv:2107.06307 tab. I]
  [arxiv:2107.06307 tab. II]. Card: [HDMapNet: An Online HD Map Construction and Evaluation
  Framework](/papers/2107.06307).
- **Volumetric transformers.** nnFormer is a 3D U-shaped network that interleaves convolution and
  self-attention, attends locally in the encoder and globally in the bottleneck, and uses skip
  attention instead of concatenation [arxiv:2109.03201 §1] [arxiv:2109.03201 §3]. Against an
  nnUNet rerun on the brain-tumour task it gives HD95 4.42 mm vs 4.60 and DSC 82.02 vs 81.87
  [arxiv:2109.03201 tab. 5]. Card: [nnFormer: Interleaved Transformer for Volumetric
  Segmentation](/papers/2109.03201).
- **LiDAR occupancy for an air-ground robot.** OccMamba assigns a class to every cell of a
  256 × 256 × 32 voxel grid built from a LiDAR point cloud. It has a semantic branch and a geometric
  branch, both with Mamba blocks, fused in bird's-eye view [arxiv:2408.10618 §3.1]. On the hidden
  SemanticKITTI test set it reports IoU 59.9, mIoU 25.0 and 22.1 FPS [arxiv:2408.10618 tab. 2]. On
  validation it reports IoU 58.6 and mIoU 25.2 with 23.8 M parameters, against 149.6 M for MonoScene
  [arxiv:2408.10618 tab. 3]. Removing the geometric or the semantic Mamba blocks gives validation
  mIoU 24.7 and 24.1 [arxiv:2408.10618 tab. 4]. The text gives different gains from these tables, so
  use the tables [arxiv:2408.10618 §5.2]. Card: [OMEGA: Efficient Occlusion-Aware Navigation for
  Air-Ground Robot in Dynamic Environments via State Space Model](/papers/2408.10618).

### Point-cloud segmentation with continuous convolutions (PCCN)

Points from a lidar or depth sensor sit unevenly in space, so a grid convolution does not apply
directly. PCCN defines the kernel as a small MLP over relative 3D offsets, which lets one layer
take scattered inputs and produce outputs at any location [arxiv:2101.06742 §3.1] [arxiv:2101.06742
§3.2]. Its locality window is the K nearest neighbours or a radius [arxiv:2101.06742 §3.3]. The
kernel is factorised so that only one kernel value per output channel is stored for each point
pair [arxiv:2101.06742 §3.3].

Author-reported, the indoor model reaches 58.27 mIoU on the Stanford indoor set against 48.92 for
SEGCloud and 41.09 for PointNet [arxiv:2101.06742 tab. 1]. On in-house driving scans, the
combination of a trained 3D-FCN with seven residual PCCN layers reaches 58.06 mIoU, against 49.28
for 3D-FCN alone; the PCCN-only model is smaller (9.34 MB) but scores lower on mIoU, 46.35
[arxiv:2101.06742 tab. 2]. The driving and flow data are in-house and not stated to be released,
and no official code was found, so these numbers cannot be checked from public sources
[arxiv:2101.06742 §4.2]. The paper's own dataset sizes disagree, so do not quote one as fact
[arxiv:2101.06742 §1] [arxiv:2101.06742 §4]. Card: [Deep Parametric Continuous Convolutional Neural
Networks](/papers/2101.06742). See also [continuous convolution](/concepts/continuous-convolution).

### Sparse LiDAR sweeps with a scene-completion teacher (JS3C-Net)

JS3C-Net labels the points of one LiDAR sweep. A sparse-voxel U-Net gives each point a
class score and a shape embedding [arxiv:2012.03762 §3.2]. A second, scene-completion decoder
predicts a dense voxel volume of the whole scene, and its targets come from voxels built from 70
merged sweeps [arxiv:2012.03762 §3.3] [arxiv:2012.03762 §4.1]. A point-voxel interaction module
links the two branches. The completion branch is used in training only [arxiv:2012.03762 §3.4]
[arxiv:2012.03762 §3.6]. The two losses are combined with learned uncertainty weights
[arxiv:2012.03762 §3.5].

Author-reported, the segmentation model reaches 66.0 mIoU on the SemanticKITTI benchmark, against
61.8 for the authors' own SparseConv baseline and 58.8 for KPConv [arxiv:2012.03762 tab. 1]. On
SemanticPOSS it reaches 60.2 average IoU, against 57.2 and 55.2 [arxiv:2012.03762 tab. 2]. On the
validation split, dropping the joint learning lowers segmentation mIoU from 67.5 to 63.1
[arxiv:2012.03762 tab. 4]. The latency table lists 471 ms for the segmentation backbone alone, and a
second row with 107 ms and 0.4 M more parameters for the version with the auxiliary and interaction
modules [arxiv:2012.03762 tab. 5]. The released code differs from the paper in its input feature
(remission, not coordinates) and in feeding raw logits to the completion branch, so check the card
before reproducing [code: kitti_dataset.py#L161-L164] [code: train.py#L116-L119]. Card:
[JS3C-Net](/papers/2012.03762). See also [semantic scene completion](/concepts/semantic-scene-completion).

### Semi-supervised LiDAR mixing with camera distillation (LaserMix, LaserMix++)

LaserMix cuts each LiDAR scan into inclination bands and swaps alternate bands between two scans,
together with their labels, to make new training scans [arxiv:2405.05258 §3.2] [arxiv:2405.05258 eq. 7].
In the semi-supervised setup, a teacher updated as a moving average of the student pseudo-labels
unlabelled scans above a confidence threshold [arxiv:2405.05258 §3.3]. LaserMix++ adds image-feature
distillation into the LiDAR branch and a CLIP-aligned loss [arxiv:2405.05258 eq. 13]
[arxiv:2405.05258 eq. 15], combined in [arxiv:2405.05258 eq. 16]. Author-reported mIoU on SemanticKITTI
with Cylinder3D and 1% labels is 45.4 for the supervised baseline, 50.6 with LaserMix and 56.2 with
LaserMix++ [arxiv:2405.05258 tab. 2]. Labelled fractions run from 1% to 50% on eight A100 GPUs
[arxiv:2405.05258 §5.1] [arxiv:2405.05258 §5.2]. The method needs calibrated, synchronised camera and
LiDAR data [arxiv:2405.05258 §4.1]. The camera-side parts of LaserMix++ have no released code, and the
image model is not named [arxiv:2405.05258 §4.4]. Results are single runs with no variance
[arxiv:2405.05258 §5.4], and the LiM3D comparison uses test-time ensembles, so it is not like for like
[arxiv:2405.05258 §5.3]. Card: [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous
Driving](/papers/2405.05258). See also [semi-supervised learning](/concepts/semi-supervised-learning).

### One encoder, three driving heads (YOLOP)

YOLOP shares one encoder across three driving tasks: vehicle detection, drivable-area segmentation
and lane-line segmentation [arxiv:2108.11250 §3] [arxiv:2108.11250 §3.2]. The encoder is CSPDarknet
with SPP and FPN [arxiv:2108.11250 §3.1]. The detection head is a YOLOv4-style single-stage
detector on three scales [arxiv:2108.11250 §3.2.1]. Both segmentation heads read the stride-8 FPN
map and upsample to full resolution [arxiv:2108.11250 §3.2.2]. The paper trains end to end by
default; a three-stage schedule is also tested, and Algorithm 1 moves to the next stage when the
loss falls below a threshold [arxiv:2108.11250 §3.4] [arxiv:2108.11250 alg. 1].

On BDD100K validation, author-reported, the single joint model reaches 89.2 recall and 76.5 mAP50
for vehicles, 91.5 mIoU for drivable area, and 70.50 lane accuracy with 26.20 IoU
[arxiv:2108.11250 tab. 1] [arxiv:2108.11250 tab. 2] [arxiv:2108.11250 tab. 3]. It runs at 41 FPS on
a TITAN XP [arxiv:2108.11250 tab. 1]. Table 5 compares the joint model with single-task models:
the joint drivable-area mIoU is 91.5 against 92.0 single-task, and the joint lane accuracy is 70.5
against 79.6 for a lane-only model [arxiv:2108.11250 tab. 5]. The joint model's time is 24.4 ms;
the three single-task times sum to 45.3 ms, a sum the card computes from that table
[arxiv:2108.11250 tab. 5]. For lanes, ENet is much faster, at 100 FPS [arxiv:2108.11250 tab. 3]. Card: [YOLOP: You Only Look Once for
Panoptic Driving Perception](/papers/2108.11250). See also [multi-task learning](/concepts/multi-task-learning)
and [object detection](/concepts/object-detection).

### Narrower models: satellite time series, a survey and polyps

Several papers here are narrower in scope. A recurrent convolutional encoder turns
Sentinel-2 time series into per-pixel crop labels for 17 classes, reaching 89.7% overall accuracy in
2016 and 89.5% in 2017 [arxiv:1802.02080 tab. 2] [arxiv:1802.02080 §5.2]. The study trains and tests
on both seasons, so it does not show transfer to an unseen year [arxiv:1802.02080 §4]
[arxiv:1802.02080 §5.2]. Card: [Multi-Temporal Land Cover Classification with Sequential Recurrent
Encoders](/papers/1802.02080). A survey, [Transformers in Remote Sensing](/papers/2209.01206),
collects reported segmentation numbers, for example 91.0 overall accuracy on Potsdam for Trans-CNN
[arxiv:2209.01206 tab. 4], but does not re-run any method [arxiv:2209.01206 §5] [arxiv:2209.01206 §6].

For colonoscopy, a benchmark on Kvasir-SEG compares detectors and segmenters on speed and accuracy.
The proposed ColonSegNet, a small residual encoder-decoder with squeeze-and-excitation blocks,
reaches Dice 0.8206 at 182.38 FPS, while the best Dice in the table is UNet with ResNet34 at 0.8757
and 35.00 FPS [arxiv:2011.07631 tab. V]. Card: [Real-Time Polyp Detection, Localization and
Segmentation in Colonoscopy Using Deep Learning](/papers/2011.07631).

HarDNet-MSEG targets the same polyp task with a HarDNet68 encoder and a light decoder. The decoder
uses RFB blocks and dense aggregation, and drops the highest-resolution encoder level
[arxiv:2101.07172 §3.1] [arxiv:2101.07172 §3.2]. Under the Kvasir-SEG 880/120 split it reports 0.904
mean Dice at 86.7 FPS [arxiv:2101.07172 tab. 1]. Under the PraNet protocol, trained on 1450 images
from Kvasir-SEG and CVC-ClinicDB, it reports 0.912 [arxiv:2101.07172 tab. 2]. The two setups differ
in input size, optimiser and training data, so the figures are not one benchmark
[arxiv:2101.07172 §4.2]. The card lists training details the paper leaves out, including the loss
and the checkpoint rule, so check it before reproducing. Card: [HarDNet-MSEG: A Simple
Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86
FPS](/papers/2101.07172).

[SANet](/papers/2108.00882) also targets polyps, with a Res2Net encoder and a decoder that gates
shallow features with deep ones. In training it swaps colour statistics between images in LAB
space, so the input keeps its content but takes another image's colours [arxiv:2108.00882 §3.1]
[arxiv:2108.00882 alg. 1]. Its shallow attention gate multiplies early features by a ReLU of the
upsampled deep features [arxiv:2108.00882 §3.2] [arxiv:2108.00882 eq. 1]. At inference, it divides
the positive and the negative logits by the fraction of each in the image, which needs no retraining
[arxiv:2108.00882 §3.3] [arxiv:2108.00882 eq. 3].

Author-reported mean Dice on five polyp sets is 0.904 on Kvasir, 0.916 on CVC-ClinicDB, 0.753 on
ColonDB and 0.750 on ETIS. PraNet scores 0.898, 0.899, 0.712 and 0.628 on the same sets
[arxiv:2108.00882 tab. 1]. The paper reports about 72 FPS on an RTX 2080 Ti, against 64 for PraNet
[arxiv:2108.00882 §4.2]. It follows PraNet's split [arxiv:2108.00882 §4.1], which differs from the
880/120 Kvasir split behind HarDNet-MSEG's 0.904, so the two Kvasir numbers are not comparable. The
released code has no shallow attention module, so build that gate from the equations rather than
from the repository (see its card). Card: [Shallow Attention Network for Polyp Segmentation](/papers/2108.00882).

### Non-learnable prototypes as class heads (ProtoSeg)

ProtoSeg removes the learned class weights, or queries, from a segmentation head. Each class is
described by K = 10 prototypes, which are centres of clusters of training-pixel embeddings
[arxiv:2203.15102 §3] [arxiv:2203.15102 §5.1]. A pixel's class score is its best cosine similarity to
any of that class's prototypes, and the pixel takes the highest-scoring class [arxiv:2203.15102 eq. 5]
[arxiv:2203.15102 eq. 6]. The clusters come from online Sinkhorn clustering with an equal-size
constraint, and the prototypes move by a momentum average instead of gradient descent
[arxiv:2203.15102 eq. 8] [arxiv:2203.15102 eq. 10] [arxiv:2203.15102 eq. 14] [arxiv:2203.15102 alg. 1].
The training loss adds a pixel-prototype contrastive term and a compactness term to cross-entropy
[arxiv:2203.15102 eq. 11] [arxiv:2203.15102 eq. 12] [arxiv:2203.15102 eq. 13].

Author-reported, the prototype head adds 0.6 to 1.2 mIoU on ADE20K over four baselines that the
authors reproduced with the MMSegmentation recipe [arxiv:2203.15102 tab. 1]. On ADE20K-847 with
SegFormer, it reaches 19.4 mIoU with 27.37 M parameters, against 17.7 for the parametric head with
33.96 M; the table counts learnable parameters only [arxiv:2203.15102 tab. 4]. In the ablation,
adding the contrastive term to cross-entropy gives 45.9 against 45.0, and a single prototype per
class already gives 45.5 [arxiv:2203.15102 tab. 5]. The released code is a different implementation
from the paper: it has no temperature, uses a lower compactness weight, and adds projection and
normalisation layers before the score, so inference is not pure nearest-prototype retrieval
[code: lib/models/nets/hrnet.py#L80-L82] [code: configs/cityscapes/H_48_D_4_proto.json#L139]. The
only released configuration is HRNet-W48 on Cityscapes, and the paper does not state how the
prototypes are initialised [arxiv:2203.15102 §5.1]. Card: [Rethinking Semantic Segmentation: A
Prototype View](/papers/2203.15102). See also [prototype learning](/concepts/prototype-learning).

### Label-free segmentation: distilling a frozen self-supervised backbone (STEGO)

STEGO uses no labels at all. It keeps a self-supervised DINO ViT-Base frozen, takes its patch
features as given, and trains a small head so that the head's per-pixel codes reproduce the
backbone's feature correspondences [arxiv:2203.08414 §3.1] [arxiv:2203.08414 §3.2]. The codes are
then clustered into classes and refined with a fully connected CRF [arxiv:2203.08414 §3.3]. The
loss is applied to three kinds of pair: an image with itself, with one of its seven nearest
neighbours, and with a shuffled batch partner [arxiv:2203.08414 §3.2] [arxiv:2203.08414 §3.3].
Clamping the codes at zero before the correlation is part of the method and not a detail; the
paper's ViT-S ablation drops unsupervised mIoU on CocoStuff-27 from 24.5 to 10.3 without it
[arxiv:2203.08414 §4.3] [arxiv:2203.08414 app. Additional Ablation Study].

Author-reported on the 27-class CocoStuff validation set, the ViT-B model reaches 28.2 unsupervised
mIoU and 41.0 linear-probe mIoU, against 14.4 and 14.8 for PiCIE + H [arxiv:2203.08414 §4.2]. On
Cityscapes-27 it reaches 21.0 unsupervised mIoU, against 12.3 for PiCIE [arxiv:2203.08414 §4.2].
The paper reports under two hours of training on one V100 [arxiv:2203.08414 §3.3]. These are
label-free numbers from a fixed 27-class ontology that the authors call partly arbitrary, so they
do not compare directly with supervised mIoU. Card: [Unsupervised Semantic Segmentation by
Distilling Feature Correspondences](/papers/2203.08414).

### Domain generalization from one source: selective whitening (RobustNet)

A segmenter trained on synthetic street scenes often degrades on real scenes with other weather,
lighting or cities. [RobustNet](/papers/2103.15597) addresses this without target images. It adds a
training-only loss to three instance-normalization layers of a DeepLabV3+ [arxiv:2103.15597 §4.4]
[arxiv:2103.15597 app. A.3]. Instance normalization standardises each channel but leaves the
correlations between channels, which the paper ties to style such as colour and blur
[arxiv:2103.15597 §1] [arxiv:2103.15597 §2]. The loss penalises only the covariance entries that move
most under colour jitter and Gaussian blur. The entries are found once, after five warm-up epochs,
and grouped with k-means at k = 3 [arxiv:2103.15597 §4.3] [arxiv:2103.15597 tab. 7].

Author-reported, trained on GTAV with ResNet-50, the method reaches 36.58, 35.20 and 40.33 mIoU on
Cityscapes, BDD-100K and Mapillary, against 28.95, 25.14 and 28.18 for the baseline
[arxiv:2103.15597 tab. 1]. The numbers are averages of three runs with no spread reported
[arxiv:2103.15597 §5.2.1]. Inference time is the same within noise (10.48 ms baseline, 10.43 ms
with the loss, one A100 at 2048 x 1024) [arxiv:2103.15597 tab. 6]. The paper does not measure
training time [arxiv:2103.15597 §5.2.3]. The released code is BSD-3-Clause, but its Cityscapes script
uses k = 2 and it adds an auxiliary loss the paper's equation omits, so check its card before
reproducing the table [arxiv:2103.15597 app. A.3]. Card: [RobustNet: Improving Domain Generalization
in Urban-Scene Segmentation via Instance Selective Whitening](/papers/2103.15597). See also
[domain generalization](/concepts/domain-generalization) and [feature whitening](/concepts/feature-whitening).

### Unsupervised domain adaptation: structure codes and entropy-free losses

When labels exist only in a synthetic source domain (GTA5), two papers adapt the segmenter to
Cityscapes without target labels.

- **DISE** splits every image into a domain-invariant structure code, used for segmentation, and a
  domain-specific texture code. It is trained with reconstruction, texture-swap translation and
  adversarial losses; a swapped image keeps the source labels and so gives the target domain extra
  supervision [arxiv:1903.12212 §3.1] [arxiv:1903.12212 §3.2]. It reaches 45.4 mIoU, against 42.4
  for the output-space adaptation of Tsai et al. [arxiv:1903.12212 tab. 2]. Its ablation runs from
  39.8 (source only) through 42.6 and 44.1 to 45.4 with label transfer [arxiv:1903.12212 tab. 4].
  Training takes 88 GPU hours on one V100 [arxiv:1903.12212 §3.3]. Card: [All about Structure:
  Adapting Structural Information across Domains for Boosting Semantic Segmentation](/papers/1903.12212).
- **Maximum squares loss** replaces the entropy penalty on unlabelled target pixels with a
  negative sum of squared probabilities [arxiv:1909.13589 §3.2] [arxiv:1909.13589 eq. 6]. Two
  additions are optional: image-wise class weights and a low-level head trained on confident
  labels [arxiv:1909.13589 §3.3] [arxiv:1909.13589 §3.4]. With DeepLabv2 and ResNet-101 it reaches
  46.4 mIoU on GTA5 to Cityscapes, against 42.2 for the authors' entropy-minimisation run
  [arxiv:1909.13589 tab. 2]. The same loss transfers to Cityscapes to NTHU cross-city adaptation,
  with 54.5 on Rome against 53.9 without the weights [arxiv:1909.13589 tab. 6]. Card: [Domain
  Adaptation for Semantic Segmentation with Maximum Squares Loss](/papers/1909.13589).

Both papers use the same benchmark and backbone family, but their baselines were partly reproduced
by the authors, so the headline numbers are not directly comparable (see Contradictions).

**Aerial imagery: translate the source images.** A different route changes the source images
instead of the network. A CycleGAN-style translator, trained without paired images, restyles
labelled Potsdam patches to the Vaihingen sensor look. A BiSeNet trained on Potsdam is then
fine-tuned on the restyled patches with their original labels [arxiv:1905.03198 §4.2]
[arxiv:1905.03198 §4.3]. Overall accuracy rises from 0.35 to 0.52, with the largest gains on Building
(0.23 to 0.71) and Tree (0.06 to 0.51) [arxiv:1905.03198 tab. 3] [arxiv:1905.03198 tab. 4]. The
paper says the other classes stay the same, but its table shows Low vegetation falling from 0.38 to
0.27 [arxiv:1905.03198 §6] [arxiv:1905.03198 tab. 4]. The best epoch is picked on the target test set,
and translation quality is checked by eye only [arxiv:1905.03198 §5.2.4] [arxiv:1905.03198 §5.2.3].
Card: [Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation
of Aerial Images](/papers/1905.03198). See also [domain adaptation](/concepts/domain-adaptation) and
[image-to-image translation](/concepts/image-to-image-translation).

### Continual segmentation: new classes, no stored images

PLOP adds classes in steps without keeping old images. It combines a multi-scale feature
distillation term (Local POD) with entropy-thresholded pseudo-labels for background pixels
[arxiv:2011.11390 §4.2]. On Pascal VOC 2012 in the 15-1 setting (fifteen
classes, then one added per step) it reaches 54.64 mIoU over all classes, against 29.70 for MiB
(excerpted from its paper) [arxiv:2011.11390 §4.2]. On ADE20K 100-50 the gap is small, 32.94 against
32.79 [arxiv:2011.11390 §4.2]. Joint training, which is not continual, reaches 77.40 on VOC
[arxiv:2011.11390 §4.2]. Card: [PLOP: Learning without Forgetting for Continual Semantic
Segmentation](/papers/2011.11390).

### Long-tail calibration of the classifier (DisAlign)

Distribution Alignment targets skewed label sets, where a few classes dominate the pixels. It
freezes the backbone and the original scores, then learns a per-class scale and offset. An
input-dependent gate blends the adjusted scores with the originals [arxiv:2103.16370 §3.2.1]
[arxiv:2103.16370 eq. 4] [arxiv:2103.16370 eq. 5]. Training uses cross-entropy with class weights
proportional to inverse frequency raised to a power rho [arxiv:2103.16370 §3.2.2]
[arxiv:2103.16370 eq. 11]. For segmentation the same formula is applied per pixel, with rho set to
0.3 in the paper [arxiv:2103.16370 §4.2] [arxiv:2103.16370 app. B.2]. The authors note that
resampling is hard to apply to segmentation and detection, while re-weighting can be used there
[arxiv:2103.16370 §3.3].

Author-reported on ADE20K, mIoU rises from 38.1 to 40.1 for FCN with ResNet-50, and from 46.4 to
47.1 for DeepLabV3+ with ResNet-101 [arxiv:2103.16370 tab. 6]. The best segmentation row is
DeepLabV3+ with ResNeSt-101, at 47.8 against 47.3 [arxiv:2103.16370 tab. 6]. The active tables
report no seeds or spread, so gains of a few tenths of a point are not established
[arxiv:2103.16370 §4]. The released segmentation config uses rho 0.2 and a 4,000-iteration
second stage, while the paper states 0.3 and 8k [code: semantic_seg/configs/ade20k_fcn_disalign/disalign_fcn_r50-d8_512x512_160k_ade20k.py#L12]
[code: semantic_seg/configs/ade20k_fcn_disalign/disalign_fcn_r50-d8_512x512_160k_ade20k.py#L22-L23].
Card: [Distribution Alignment: A Unified Framework for Long-tail Visual Recognition](/papers/2103.16370).
See also [long-tailed recognition](/concepts/long-tailed-recognition) and [class-balanced loss](/concepts/class-balanced-loss).

### Weak and open-vocabulary segmentation

- **Weak supervision from image labels (ReCAM).** ReCAM keeps a converged multi-label classifier
  and adds a second fully connected layer trained with softmax cross-entropy on single-class
  features, which are the backbone map weighted by each class's normalised CAM and then pooled
  [arxiv:2203.00962 §4.1]. On PASCAL VOC 2012 seeds it lifts CAM from 48.8 to 54.8 mIoU, and with
  IRN refinement from 66.3 to 70.9 [arxiv:2203.00962 tab. 2]. Its cost is close to the base method
  in the ResNet-50 setting (1.0 to 1.9 units) [arxiv:2203.00962 tab. 2]. Card: [Class Re-Activation
  Maps for Weakly-Supervised Semantic Segmentation](/papers/2203.00962).
- **Open vocabulary (OpenSeg).** OpenSeg represents an image as mask proposals with features, not
  as a dense map alone. It is trained with a Dice loss on class-agnostic COCO masks plus a caption
  grounding loss [arxiv:2112.12143 §3.3]. With EfficientNet-B7 and prompt engineering it reaches
  72.2 mIoU on PASCAL-20 zero-shot, against 52.3 for LSeg with a ViT-L/16 backbone
  [arxiv:2112.12143 tab. 2] [arxiv:2112.12143 §4.3]. The ADE-150 and Pascal-Context numbers are much
  lower (for example 28.6 on ADE-150 for the same model) [arxiv:2112.12143 tab. 2]. The paper has no
  official code, and some baseline rows are copied from other papers [arxiv:2112.12143 §4]. Card:
  [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143). See
  also [open-vocabulary segmentation](/concepts/open-vocabulary-segmentation).
- **Frozen CLIP with a side network (SAN).** SAN keeps CLIP's ViT frozen and attaches a light side ViT
  of 8 blocks at width 240, which also takes fused CLIP features [arxiv:2302.12242 §3]
  [arxiv:2302.12242 §4.2]. The side network outputs mask proposals and per-head attention biases.
  The biases steer CLIP's last blocks, which then classify each proposal [arxiv:2302.12242 eq. 1]
  [arxiv:2302.12242 eq. 2] [arxiv:2302.12242 eq. 3]. Training is end to end, through CLIP's
  recognition blocks, on COCO-Stuff [arxiv:2302.12242 §3] [arxiv:2302.12242 §4.1]. Author-reported
  mIoU on ADE-150 is 27.5 with ViT-B/16 and 32.1 with ViT-L/14, without the ensemble rows
  [arxiv:2302.12242 tab. 2]. At 640 x 640 with ViT-B/16 the side network adds 8.4 M trainable
  parameters and runs at 15.2 FPS, against 147.2 M and 0.8 FPS for OvSeg, which the authors
  re-implemented [arxiv:2302.12242 tab. 3]. The paper's ensemble rows use a model that is not in the
  public code, so compare the non-ensemble rows [arxiv:2302.12242 tab. 2]. Card: [Side Adapter
  Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242). Its main baseline,
  [OvSeg](/papers/2210.04150), has its own card.
- **Text-only supervision (GroupViT).** GroupViT learns segments from image-caption pairs and no
  masks. A ViT-S starts from 196 patches, and learned group tokens merge them into 64 and then 8
  segments through a hard assignment that keeps the soft gradient, trained with contrastive
  image-text losses [arxiv:2202.11094 §3.1] [arxiv:2202.11094 §3.2]. A multi-label term adds three
  nouns from each caption, placed in short templates, as extra positives [arxiv:2202.11094 §3.2]
  [arxiv:2202.11094 §4.1]. Labels are matched to the output segments by text similarity, with no
  fine-tuning. Zero-shot mIoU is 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context after training
  on about 26M public image-text pairs [arxiv:2202.11094 tab. 5]. The supervised DeiT row reports
  53.0 and 35.9 [arxiv:2202.11094 tab. 5]. The hard assignment carries much of the result: in the
  one-stage ablation, removing it drops mIoU from 39.3 to 25.1 with the multi-label term kept
  [arxiv:2202.11094 tab. 1]. Background is the weak point. On PASCAL Context, classes such as ground,
  road and wall are grouped but rarely named in captions, so their IoU is about 1.5
  [arxiv:2202.11094 app. C.3]. Card: [GroupViT: Semantic Segmentation Emerges from Text
  Supervision](/papers/2202.11094). The official code is non-commercial, so implement from the paper
  (see its card).

- **Two-stage CLIP labelling of mask proposals (zsseg).** A MaskFormer trained on seen classes
  proposes class-agnostic masks, and a frozen CLIP ViT-B/16 scores each masked crop against text
  prompts [arxiv:2112.14757 §4.1] [arxiv:2112.14757 §4.2.1]. Each mask is binarised at 0.5, the
  background is filled with a constant colour, and the crop is resized to 224 by 224
  [arxiv:2112.14757 §4.2.1]. The two classifiers are combined by default, and the per-mask scores
  are summed into a pixel map [arxiv:2112.14757 eq. 2] [arxiv:2112.14757 tab. 7]. Author-reported
  zero-shot hIoU is 37.8 on COCO-Stuff, against 20.9 for FCN, and 77.5 on Pascal VOC, against 50.7
  for FCN and 39.7 for CaGNet, the best listed prior without self-training [arxiv:2112.14757 tab. 2]
  [arxiv:2112.14757 tab. 3]. Direct CLIP classification of MaskFormer masks reaches 28.2 hIoU, against
  11.0 for Selective Search and 10.9 for GPB-UCM proposals [arxiv:2112.14757 tab. 5]. The paper names
  no ensemble weight; the released configs use 0.8 for COCO-Stuff and 0.7 for VOC
  [code: configs/coco-stuff-164k-156/zero_shot_maskformer_R50_bs32_60k.yaml#L21]. The "+ST" rows
  have no procedure in the paper, and the released evaluator is the only description
  [code: mask_former/evaluation/pseudo_sem_seg_evaluation.py#L81-L96]. Card: [A Simple Baseline for
  Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757).

### Segmentation as a generated output

FashionGAN generates a segmentation map as an intermediate step, then renders clothing texture
region by region from a text description [arxiv:1710.07346 §3.1]. The map has seven labels and is
generated from an 8x8 coarse map of the wearer that merges all clothing into one class
[arxiv:1710.07346 §3.2]. Its quality is judged indirectly: attribute detection on the generated
images reaches mAP 82.6, against 71.8 to 74.7 for the baselines and 86.8 on real photos
[arxiv:1710.07346 tab. 1]. In a user study, participants labelled the generated map as real in
42% of cases, which the authors read as near chance [arxiv:1710.07346 §4.3]. Card: [Be Your Own
Prada: Fashion Synthesis with Structural Coherence](/papers/1710.07346).

PixelLM takes the idea further: a multimodal language model answers in text and emits one mask per
named target. For each target the model writes a set of learned codebook tokens, and a small pixel
decoder turns their hidden states into a mask, fusing them across scales and gating each scale by the
mask from the coarser one [arxiv:2312.02228 §3.1] [arxiv:2312.02228 eq. 1]. Training uses text
cross-entropy, a refinement loss and a DICE term [arxiv:2312.02228 §3.2]. On the MUSE benchmark, the
7B model reaches 39.2 gIoU and 46.3 cIoU, against 12.8 and 27.1 for LISA-7B [arxiv:2312.02228 tab. 1].
LISA is higher on refCOCO val and testB [arxiv:2312.02228 tab. 3]. The model is a LoRA-tuned LLaVA with
a frozen CLIP encoder, trained for about 1.5 to 2 days on eight A100 GPUs [arxiv:2312.02228 §5.1].
MUSE was built with GPT-4V, so rebuilding it needs API access [arxiv:2312.02228 §4.2]. Card:
[PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228).

### Few-shot, in-context and promptable models

Three papers change what the user supplies at test time instead of retraining.

**Few-shot cost aggregation (VAT).** VAT builds the full correlation volume between query and
support positions, then aggregates it with a 4D convolutional Swin Transformer in a three-level
pyramid, where each coarser level guides the next finer one. A decoder with appearance cues then
produces the mask [arxiv:2207.10866 §3.3] [arxiv:2207.10866 §3.4] [arxiv:2207.10866 §3.5]. Small
overlapping 4D convolutions come before the window attention, so window edges do not cut off local
context [arxiv:2207.10866 §3.4.2]. Backbones are frozen ImageNet ResNets [arxiv:2207.10866 §4.1]. With
ResNet-101, 1-shot mIoU on PASCAL-5i is 67.9 against 66.2 for HSNet, and on FSS-1000 it is 90.3
against 86.5 [arxiv:2207.10866 tab. 1] [arxiv:2207.10866 tab. 3]. The volume is large: a standard
transformer over it needed about 84 GB per batch, against 3.8 GB for VAT's aggregator
[arxiv:2207.10866 tab. 5]. The paper says it uses no augmentation and divides the vote sum by K, but
the released configs differ (see Contradictions). Card: [Cost Aggregation with 4D Convolutional Swin
Transformer for Few-Shot Segmentation](/papers/2207.10866). See also [cost aggregation](/concepts/cost-aggregation)
and [few-shot learning](/concepts/few-shot-learning).

**In-context medical segmentation (UniverSeg).** UniverSeg segments a new medical task without any
training. A fully convolutional network takes the query image and a support set of image-label
pairs, and the two streams exchange information in each block through a cross-convolution
[arxiv:2304.06131 §3.1]. It is trained once on MegaMedical, 53 datasets, with task-level augmentation
and synthetic tasks [arxiv:2304.06131 §4] [arxiv:2304.06131 app. C] [arxiv:2304.06131 app. D]. On six
held-out datasets its mean Dice is 71.8 at 1.18 M parameters, against 41.8 to 50.1 for the few-shot
baselines and 84.4 for per-task nnU-Net [arxiv:2304.06131 tab. 1]. Dice rises with support size and
levels off near 64 pairs, and averaging predictions over several support draws helps
[arxiv:2304.06131 §5.3] [arxiv:2304.06131 fig. 7]. The cost grows with the number of support pairs.
The training code is not released, and the released model differs from the paper in depth and
resampling (see its card). Card: [UniverSeg: Universal Medical Image Segmentation](/papers/2304.06131).
See also [in-context learning](/concepts/in-context-learning).

**Promptable 3D segmentation (SAM-Med3D).** SAM-Med3D applies the promptable design of SAM to whole
volumes. It has a 3D ViT image encoder, a point-prompt encoder and a 3D mask decoder, trained from
scratch in two stages [arxiv:2310.15161 §4.1] [arxiv:2310.15161 §4.2]. One click yields a 3D mask.
Author-reported overall Dice is 76.27% with one 3D point, against 36.83% for SAM-Med2D prompted on
each slice, and 80.71% with ten points [arxiv:2310.15161 tab. 3]. In the authors' pre-experiment,
training from scratch is the most balanced across seen and unseen targets (55.79 and 34.63), against
69.13 and 18.43 for fine-tuning SAM [arxiv:2310.15161 tab. 2]. The release does not include the
paper's sliding-window inference for large volumes, and its training differs from the text (see its
card) [arxiv:2310.15161 §4.3]. Card: [SAM-Med3D: Towards General-purpose Segmentation Models for
Volumetric Medical Images](/papers/2310.15161). See also [promptable segmentation](/concepts/promptable-segmentation).

### Self-supervised per-pixel features (VADeR)

VADeR is a pre-training method, not a segmenter. It learns one feature per pixel without labels. Two
augmented views of an image give known pixel correspondences through their crop and resize geometry,
and a contrastive loss pulls matched pixels together against a queue of pixels from other images
[arxiv:2011.05499 §3.1] [arxiv:2011.05499 §3.2]. The encoder starts from MoCo v2 and the decoder from
random weights [arxiv:2011.05499 §3.2]. With a linear 1x1 head on frozen features, the authors report
56.7 mIoU on VOC, against 54.4 for ImageNet-supervised ResNet-50 and 43.0 for MoCo
[arxiv:2011.05499 tab. 1]. On Cityscapes the same frozen features score 44.3, against 47.1 for
supervised pre-training, so the VOC lead does not carry over to every dense task
[arxiv:2011.05499 tab. 1]. The tables average five trials without standard deviations
[arxiv:2011.05499 §4.1]. No official code was found (see its card). Card: [Unsupervised Learning of
Dense Visual Representations](/papers/2011.05499). See also [self-supervised learning](/concepts/self-supervised-learning).

### Diffusion models: the mask as a denoising chain

MedSegDiff generates the mask with a diffusion probabilistic model. The reverse chain starts from
Gaussian noise in the mask channel and denoises it in steps, conditioned on the scan
[arxiv:2211.00611 §2] [arxiv:2211.00611 eq. 1]. Two additions carry the method. Dynamic conditional
encoding fuses the current noisy mask into the image encoder at every step
[arxiv:2211.00611 §2.1] [arxiv:2211.00611 eq. 3], and FF-Parser is a learned Fourier-domain filter on
the skip features [arxiv:2211.00611 §2.2] [arxiv:2211.00611 eq. 4]. Inference draws 25 samples of
100 steps each and fuses them with STAPLE [arxiv:2211.00611 §3.2], which is many network evaluations
per image.

Author-reported Dice for MedSegDiff++ is 87.5 on optic cup, 90.5 on brain tumour and 86.6 on
thyroid nodule. In the same table nnUNet reaches 84.9, 88.2 and 84.2 [arxiv:2211.00611 tab. 1]. The
ablation credits dynamic conditioning with 2.1, 1.6 and 1.8 Dice points and FF-Parser with a further
0.8, 0.7 and 0.7, on the same three tasks [arxiv:2211.00611 tab. 2]; the model size used for the
ablation is not stated. The paper's text gives different gains in places (see Contradictions). The
official code differs from the paper in its defaults and in parts of the architecture, so a
reimplementation from the paper alone will not match the reported numbers [arxiv:2211.00611 §3.2].
Card: [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](/papers/2211.00611).

[SegDiff](/papers/2112.00390) is a diffusion model for masks on street scenes, aerial images and
microscopy. Its noise predictor passes the noisy mask through one convolution and the image through a
separate residual dense encoder, adds the two feature maps, and feeds the sum to a U-Net
[arxiv:2112.00390 §4] [arxiv:2112.00390 eq. 17]. It is trained from random initialisation, with no
pretrained backbone [arxiv:2112.00390 §5]. The mask is the average of several independently sampled
reverse chains, then thresholded, and the paper uses 30 chains in its main experiments
[arxiv:2112.00390 §4.1]. Cityscapes is scored per object crop, not per image [arxiv:2112.00390 §5].

Author-reported: mIoU 76.44 on Cityscapes at 15% box expansion, against 73.70 for Spline-GCN
[arxiv:2112.00390 tab. 1]; F1 95.14 on Vaihingen buildings, against 94.80 for Deep contour
[arxiv:2112.00390 tab. 2]; Dice 81.59 on MoNuSeg, against 79.55 for MedT [arxiv:2112.00390 tab. 3].
The chain count is the main cost, since each chain runs a full set of reverse steps. The released code
hard-codes 9 chains and differs from the paper's noise schedule, and it has no LICENSE file, so its
code cannot be reused (see its card). Card: [SegDiff: Image Segmentation with Diffusion Probabilistic
Models](/papers/2112.00390).

A diffusion model can also act as a fixed prior at inference time. Graikos et al. train a denoiser
independently of the task and add a differentiable constraint while sampling. For land cover the
constraint has two terms: mutual information between predicted labels and a local colour clustering,
and a KL divergence to weak labels in blocks [arxiv:2206.09012 §4] [arxiv:2206.09012 app. B.3]. Training
uses Pittsburgh patches, and the other EnviroAtlas cities are the test sets
[arxiv:2206.09012 tab. 1]. On Durham the fine-tuned variant reaches 79.8% accuracy and 46.4 IoU,
against 78.9% and 47.9 for a supervised model trained on weak labels. On Phoenix the implicit-posterior
baseline is ahead on both metrics (76.2% and 46.0, against 69.6% and 32.4) [arxiv:2206.09012 tab. 1].
The claim is therefore comparable accuracy, not uniformly better results. Card: [Diffusion models as
plug-and-play priors](/papers/2206.09012).

A pretrained diffusion model can also serve as a feature extractor for a pixel classifier. Each
labelled image is noised to steps 50, 150 and 250, and the decoder activations of five UNet blocks
are upsampled to image size and concatenated into 8448 values per pixel [arxiv:2112.03126 §3.2].
An ensemble of ten small MLPs is trained on the pixels of the few annotated images, and each test
pixel takes the majority vote of the ensemble [arxiv:2112.03126 §3.2] [arxiv:2112.03126 app. C].
The authors find later diffusion steps more semantic and middle decoder blocks most informative, and
they keep the same steps and blocks for every dataset [arxiv:2112.03126 §3.1] [arxiv:2112.03126 §3.2].

Author-reported mean IoU, averaged over five data splits, is the highest in every column of the main
table: 49.4 on Bedroom-28 against 47.9 for DatasetDDPM and 45.0 for MAE, and 59.1 on FFHQ-34
against 58.8 for MAE [arxiv:2112.03126 tab. 2]. The margin over MAE is small on FFHQ-34 (59.1 against
58.8) and Cat-15 (53.7 against 52.4) [arxiv:2112.03126 tab. 2]. With fewer labels the score drops, to 38.2 on
Bedroom-28 with 10 images against 49.4 with 40 [arxiv:2112.03126 tab. 4]. The method needs a
diffusion model trained on the target domain, which the authors expect to be hard for complex
domains such as ImageNet or MSCOCO [arxiv:2112.03126 §5]. Each image yields 256 x 256 x 8448
float32 values, which is about 2.2 GB by calculation from the stated sizes, so feature storage is
the practical bottleneck [arxiv:2112.03126 §3.2]. The paper says each MLP trains for about four
epochs, which the released code does not do (it stops early), so the training recipe must be fixed
by the implementer [arxiv:2112.03126 app. C]. Card: [Label-Efficient Semantic Segmentation with
Diffusion Models](/papers/2112.03126). See also [few-shot learning](/concepts/few-shot-learning).

### Semantic SLAM with Gaussian splats

GS3LAM keeps a dense semantic map inside a set of 3D Gaussians. Each Gaussian carries a position, a
shape, a colour, an opacity and a low-dimensional semantic feature, which a learned layer decodes to
class scores with a softmax [arxiv:2603.27781 §3.2] [arxiv:2603.27781 eq. 8]. Tracking fits one camera
pose per frame with the map frozen. Mapping refines the map on the current frame and on randomly drawn
keyframes [arxiv:2603.27781 §3.4] [arxiv:2603.27781 §3.3.3]. A scale regulariser keeps Gaussian sizes
in range. The paper names it as a contribution, but the released Replica config switches it off (see
its card) [arxiv:2603.27781 §3.3.2]. Author-reported on Replica: rendering PSNR 36.26 over eight
scenes [arxiv:2603.27781 tab. 2], mIoU 96.63 over four scenes [arxiv:2603.27781 tab. 4], and tracking
ATE 0.37 cm, against 0.36 cm for SplaTAM [arxiv:2603.27781 tab. 3]. Card: [GS3LAM: Gaussian Semantic
Splatting SLAM](/papers/2603.27781). See also [SLAM](/concepts/simultaneous-localization-and-mapping)
and [Gaussian splatting](/concepts/gaussian-splatting).

### Distilled feature fields in 3D Gaussians (Feature 3DGS)

Feature 3DGS gives each 3D Gaussian a feature vector and distils it from a 2D teacher, LSeg or SAM,
so that text and prompts can select regions in an offline 3D scene [arxiv:2312.03203 §3.2]. Colour and
feature are rendered together, and the loss adds an L1 feature term to the RGB loss with weight 1.0
[arxiv:2312.03203 eq. 4]. An optional speed-up renders fewer feature channels and lifts them back with a
learned 1x1 convolution [arxiv:2312.03203 §3.2]. A text query is matched to features by cosine
similarity and a softmax over labels [arxiv:2312.03203 eq. 5] [arxiv:2312.03203 eq. 6]. On Replica,
author-reported mIoU is 0.782 at 14.55 FPS with the speed-up and 0.787 at 6.84 FPS without it, against
0.636 at 5.38 FPS for NeRF-DFF [arxiv:2312.03203 tab. 2]. The paper says the speed-up does not affect
performance [arxiv:2312.03203 §3.2], but its own dimension sweep drops LSeg features to mIoU 0.354 at
dimension 8 [arxiv:2312.03203 tab. A]. The quantitative results are on Replica only. The code licence
is non-commercial research use only, and the card records it as such. Card: [Feature 3DGS: Supercharging
3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203). See also
[Gaussian splatting](/concepts/gaussian-splatting).

### LiDAR semantic SLAM: labels decide which surfels to trust (SuMa++)

SuMa++ adds per-point semantics to a surfel-based LiDAR mapping pipeline. A segmenter labels each
point of the scan's spherical projection, and the labels are used at three points: a flood-fill
cleans the label image, the map update penalises surfels whose stored class conflicts with a
movable class, and the ICP residuals are weighted by semantic compatibility [arxiv:2105.11320 §III-C]
[arxiv:2105.11320 §III-D] [arxiv:2105.11320 §III-E] [arxiv:2105.11320 §III-F]. Deleting every
movable surfel instead is not a fix: parked cars help registration, and on urban sequences the
deletion variant diverges [arxiv:2105.11320 §IV-B] [arxiv:2105.11320 tab. II].

Author-reported, on KITTI road sequences 30 to 41 the average relative translation error falls from
6.13% (SuMa) to 1.46% (SuMa++), and on odometry training sequences 00 to 10 it falls from 0.83% to
0.70% [arxiv:2105.11320 tab. I] [arxiv:2105.11320 tab. II]. On sequence 35, a highway, the error falls
from 26.8% to 1.11% [arxiv:2105.11320 tab. I]. The training-sequence numbers are in-sample, because
the segmenter was trained on sequences 00 to 10 except 08 [arxiv:2105.11320 §IV]. The method needs
a labelled training set and a trained segmenter, and the evaluation is on KITTI only
[arxiv:2105.11320 §IV] [arxiv:2105.11320 §V]. Card: [SuMa++: Efficient LiDAR-based Semantic
SLAM](/papers/2105.11320). See also [SLAM](/concepts/simultaneous-localization-and-mapping) and
[iterative closest point](/concepts/iterative-closest-point).

### Distillation for dense prediction

DIST replaces the exact KL match in knowledge distillation with a Pearson-correlation loss, applied
across classes within each instance and across instances within each class [arxiv:2205.10536 §3.1]
[arxiv:2205.10536 §3.2]. On Cityscapes with a DeepLabV3-R18 student and an R101 teacher it reaches
77.10 mIoU, against 74.21 for the student alone and 75.42 for SKD [arxiv:2205.10536 tab. 7]. On
ImageNet the gain is largest with the strongest teacher tested (ResNet-18 student: +0.86 with ResNet-34, +1.12 with
ResNet-152) [arxiv:2205.10536 tab. 3]. Card: [Knowledge Distillation from A Stronger
Teacher](/papers/2205.10536). See also [knowledge distillation](/concepts/knowledge-distillation).

### Building footprints from satellite imagery

Continent-scale building mapping uses a U-Net with a ResNet-50-v2 encoder and residual decoder
blocks. Training combines an edge-weighted cross-entropy with a focal Tversky term, and inference
thresholds the averaged confidence and takes connected components as instances
[arxiv:2107.12283 §4] [arxiv:2107.12283 §5] [arxiv:2107.12283 §10]. Mixup on the cross-entropy term,
ImageNet initialisation and soft-label self-training are the other main additions
[arxiv:2107.12283 §6] [arxiv:2107.12283 §7] [arxiv:2107.12283 §8]. Author-reported, mixup and edge
weighting have the largest ablation effects, and self-training adds +0.057 mAP on average
[arxiv:2107.12283 §11] [arxiv:2107.12283 fig. 8]. The pipeline produced 516M polygons after
deduplication [arxiv:2107.12283 §12]. Card: [Continental-Scale Building Detection from High
Resolution Satellite Imagery](/papers/2107.12283). See also [instance segmentation](/concepts/instance-segmentation).

### Infrared small targets: nested skips and one centroid per target (DNA-Net)

Single-frame infrared targets are often one to a few tens of pixels wide, and pooling can remove a
one-pixel target before the output stage [arxiv:2106.00487 §1] [arxiv:2106.00487 §3.2.1]. DNA-Net
nests U-shaped sub-networks densely. Each node takes earlier nodes at its own level, the level below
in the previous column, and the level above in its own column [arxiv:2106.00487 eq. 2]
[arxiv:2106.00487 §3.2]. Each node block ends with channel and then spatial attention, and the
output fuses the top nodes of all five levels [arxiv:2106.00487 §3.2.3] [arxiv:2106.00487 eq. 7]
[arxiv:2106.00487 §3.3]. Connected components of the thresholded mask are reduced to centroids
[arxiv:2106.00487 §3.4] [arxiv:2106.00487 eq. 8].

On NUDT-SIRST, a synthetic set of 1327 images, the ResNet-18 variant reports IoU 87.09, Pd 98.73 and
Fa 4.223 (IoU and Pd in units of 10^-2, Fa in units of 10^-6), against IoU 81.40, Pd 96.51 and Fa
9.261 for ALCNet [arxiv:2106.00487 tab. II]. On NUAA-SIRST, a real set, it reports the best Pd and
Fa, while its IoU of 77.47 is second to the ResNet-34 variant at 77.54 [arxiv:2106.00487 tab. II].
Pd counts matched targets under a 3-pixel centroid tolerance, Fa counts false-alarm pixels, and the
reported Pd moves with that tolerance: 98.51 at 2 pixels and 98.73 at 3 [arxiv:2106.00487 §5.1] [arxiv:2106.00487 tab. IV].
Every CNN baseline was retrained by the authors on the same split, so the numbers are not the
baselines' published results [arxiv:2106.00487 §5.3]. Training uses Adagrad with a learning rate of
0.05, batch 16 and 1500 epochs [arxiv:2106.00487 §5.2]. The paper does not give the probabilities
that choose target types, or the full blur sigma list, for the synthetic data [arxiv:2106.00487 §4.2],
so that data cannot be regenerated from the paper alone. Card: [Dense Nested Attention Network for
Infrared Small Target Detection](/papers/2106.00487).

### Datasets, benchmarks and surveys

Synscapes is a synthetic street-scene set of 25,000 path-traced images. Each image has a
Cityscapes-style class map, an instance map, a depth map and per-actor metadata
[arxiv:1810.08705 §3] [arxiv:1810.08705 §3.2]. Scenario parameters are sampled independently for each
image, which allows error to be measured against distance, occlusion and orientation
[arxiv:1810.08705 §6]. Author-reported, a Cityscapes-trained DeepLab v3+ scores 63.64 mean IoU on the
Synscapes validation split, against 45.84 on the GTA-based set [arxiv:1810.08705 tab. 1].
Pre-training on Synscapes and then fine-tuning on Cityscapes gives 78.85, against 76.56 for
Cityscapes alone [arxiv:1810.08705 tab. 4]. The authors attribute the gap to realism, but the
GTA-based set is the most varied in models and building types, so realism is not isolated
[arxiv:1810.08705 §7]. The paper's CC BY licence covers the paper, not the dataset, and the paper
does not state the dataset's licence or download location [arxiv:1810.08705 §3]. Card:
[Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](/papers/1810.08705). See
also [sim-to-real transfer](/concepts/sim-to-real-transfer).

Replica is a set of 18 indoor 3D scenes with dense meshes, 16-bit HDR textures, semantic labels in
88 classes and mirror and glass reflectors [arxiv:1906.05797 §4] [arxiv:1906.05797 tab. 1]. It is a
data release, not a method, and the paper reports no learning benchmark, so it does not show that
models trained on it transfer to real images [arxiv:1906.05797 abstract]. The paper is CC BY 4.0,
but the scene data is covered by the repository's research terms, which allow only non-commercial
research or education. Check those terms before any commercial use. Card: [The Replica Dataset: A
Digital Replica of Indoor Spaces](/papers/1906.05797). See also [robot simulation](/concepts/robot-simulation).

RELLIS-3D is an off-road dataset with image and LiDAR labels in 20 classes. The authors report 51.55
mIoU for HRNet+OCR and 52.92 for Gated-SCNN on images, and 43.07 for SalsaNext and 19.97 for KPConv
on point clouds [arxiv:2011.12954 tab. 2]. **Licensing:** the card records that the repository states
non-commercial terms for the data and the code. Do not use this dataset or its code in a commercial
context without the authors' permission. Card: [RELLIS-3D Dataset](/papers/2011.12954).

A benchmark for 3D medical segmentation, [nnU-Net Revisited](/papers/2404.09556), trains CNN,
transformer and Mamba models on six public datasets (BTCV, ACDC, LiTS, BraTS2021, KiTS2023 and
AMOS2022). Each model is trained from scratch on one 40 GB A100, with five-fold cross-validation on
nnU-Net's splits [arxiv:2404.09556 §3.1] [arxiv:2404.09556 §3.2]. The authors argue that many
architecture gains come from confounds such as extra data, pre-training, a weaker baseline or a
small split [arxiv:2404.09556 §2.1] [arxiv:2404.09556 §2.2]. Under this protocol, the CNN U-Nets
built in the nnU-Net framework lead. MedNeXt L with kernel size 5 has the best mean Dice on BTCV,
LiTS, BraTS and AMOS, MedNeXt L with kernel size 3 on ACDC, and nnU-Net ResEnc XL on KiTS2023
[arxiv:2404.09556 tab. 1]. The transformer models (SwinUNETR, nnFormer and CoTr) do not match the
original nnU-Net, and the U-Mamba gain over the "No-Mamba Base" ablation disappears; the card
attributes the earlier gain to the residual encoder [arxiv:2404.09556 §4] [arxiv:2404.09556 tab. 1].
The dataset suitability ratio, inter-method over intra-method spread, is highest for AMOS, KiTS and
ACDC when all methods are included, and it falls for BTCV and ACDC without the Auto3DSeg runs
[arxiv:2404.09556 tab. 3]. The ranking therefore depends on the method set. Only the nnU-Net code
was checked; the other methods' code was not [arxiv:2404.09556 §3.1]. Card: [nnU-Net Revisited: A
Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556).

A survey of lane topology for HD maps, [Yao et al.](/papers/2504.01989), uses segmentation as one step
in its aerial route. A segmentation network labels road pixels, and morphological post-processing then
turns the mask into a road graph [arxiv:2504.01989 §2.2]. The paper does not run any method itself, and
its metrics are road-graph scores (APLS, TLTS) rather than mIoU [arxiv:2504.01989 §3.2]. It reports the
aerial route's weakness as occlusion by buildings and trees, and it calls aerial methods unsuitable for
automatic HD-map construction [arxiv:2504.01989 §2.3].

The anomaly-detection survey is not a segmentation benchmark and reports no experiments of its own.
Its camera section lists confidence scores from segmentation networks, such as Bayesian SegNet and
standardized max logits, among the baselines for unusual road scenes [arxiv:2204.07974 §2]. Pixel-level
anomalies are outside its scope [arxiv:2204.07974 §1]. Card: [Anomaly Detection in Autonomous Driving:
A Survey](/papers/2204.07974).

A review, [Deep Semantic Segmentation of Natural and Medical Images](/papers/1910.07655), groups deep
segmentation work into architectures, losses, synthesis, weak supervision and multi-task models. It runs
no experiments of its own [arxiv:1910.07655 §1] [arxiv:1910.07655 §10]. Its loss section is the most
practical part. In a worked example with one false negative, Dice gives 0.96 on a large object and 0.66
on a small one, while cross-entropy gives 0.83 for both. The review advises pairing overlap
losses with cross-entropy [arxiv:1910.07655 §4.4] [arxiv:1910.07655 §10.3]. These values are
illustrative, not benchmark results. Its VOC 2012 mean-IoU table collects numbers from other papers,
which use different backbones and protocols, so it is not a ranking [arxiv:1910.07655 tab. 2].

Three more surveys cover segmentation within a wider scope. A review of deep learning on UAV
imagery codes 232 articles and proceedings from 2016 to 2021 by network, task, sensor and context.
Its tallies put object detection at 53.9% and segmentation at 40.7% of the studies
[arxiv:2101.10861 §3]. It runs no experiments and its percentages are the authors' own
[arxiv:2101.10861 §3]. Card: [A Review on Deep Learning in UAV Remote Sensing](/papers/2101.10861).

A survey of deep learning for weed detection covers 70 papers from 2016 to August 2020 and lists 19
public datasets, most of them specific to one site or crop [arxiv:2103.01415 §4] [arxiv:2103.01415 §6.3].
Its accuracy figures come from different crops, weed species and metrics, and the survey does not
rank them [arxiv:2103.01415 §11]. Card: [A Survey of Deep Learning Techniques for Weed Detection from
Images](/papers/2103.01415).

A review of vision transformers in medical imaging groups segmentation models by where the
transformer sits in the network [arxiv:2301.03505 §2.2]. Its Synapse table lists MERIT at 84.90
Dice, but the review does not say whether each row was copied or re-run, so the rows may not share a
protocol [arxiv:2301.03505 tab. 6]. Card: [Advances in Medical Image Analysis with Vision
Transformers: A Comprehensive Review](/papers/2301.03505).

### Augmentation survey with segmentation runs

The survey sorts image augmentation into manipulation, erasing, mixing, automated policy search,
feature-space and GAN-based methods, and notes that the label must change with the image
[arxiv:2204.08610 §1] [arxiv:2204.08610 §2.1]. Experiments cover only the first three groups
[arxiv:2204.08610 §4.1]. On PASCAL VOC, author-reported mean IoU moves from 75.32 to 75.75 for
DeepLabV3+ and from 71.65 to 74.26 for ISANet [arxiv:2204.08610 tab. 2]. Each result is a single run
with no variance [arxiv:2204.08610 §4]. The printed change column disagrees with the recomputed
differences in several rows, so recompute from the two value columns [arxiv:2204.08610 tab. 2]
[arxiv:2204.08610 tab. 4]. No code is released, and the optimiser, epochs, batch size and segmentation
backbone are not stated [arxiv:2204.08610 tab. 1]. Card: [Image Data Augmentation for Deep Learning: A
Survey](/papers/2204.08610).

### Mid-level labels rendered from 3D scans (Omnidata)

Omnidata is a pipeline that renders mid-level vision labels from textured 3D scans. It places cameras
by Poisson-disc sampling, samples points, and keeps only points seen by at least three cameras
[arxiv:2110.04994 §3.1]. It renders 21 cues, including depth and surface normals, each taking 1 to 4
seconds per 512 x 512 image on a CPU [arxiv:2110.04994 §3.1]. The starter set has 14,601,449 images
from 2,414 scenes [arxiv:2110.04994 §4.2] [arxiv:2110.04994 tab. 1]. Zero-shot depth on NYU reaches an
L1 error of 0.2878, against 0.3838 for MiDaS v3 [arxiv:2110.04994 tab. 2]. The full cue pipeline is
only in the supplement, and the 21 cues are shown in a figure only [arxiv:2110.04994 §3.1]. The
starter dataset and weights are for non-commercial research and education, and the card records that
licence. Card: [Omnidata](/papers/2110.04994).

### Scope note: a driving agent that takes segmentation images (InDRiVE)

InDRiVE is listed here because its input is a semantic-segmentation image, not because it segments.
It pretrains a DreamerV3-style world model in CARLA with ensemble disagreement as an intrinsic reward
and no task reward, then tests it zero-shot on an unseen town [arxiv:2512.18850 §3.2.1]
[arxiv:2512.18850 §4.1]. The card gives zero-shot success rates of 75.65 on Town01 and 73.20 on
Town02 [arxiv:2512.18850 tab. 6]. The repository licence is non-commercial, apart from the dreamerv3
folder, so the code is not commercial-safe. Card:
[InDRiVE](/papers/2512.18850).

### Anomaly detection by resynthesis

Unknown objects on a road are often given a known label by a segmenter. The resynthesis approach
repaints the image from the predicted label map with a pix2pixHD generator, then scores the pixels
where the repainting disagrees with the photo [arxiv:1904.07595 §3] [arxiv:1904.07595 fig. 2]. A
discrepancy network reads the photo, the repainting and the labels. It is trained only on known
classes: a random subset of object instances is given a different known label, the generator repaints
from that altered map, and the changed pixels become the positive target [arxiv:1904.07595 §3.2]. The
released config uses half the instances (see the card). Author-reported AUROC is 0.79 to 0.93 on Lost
and Found and a 60-image road set, above a Bayesian-uncertainty baseline and an RBM reconstruction
baseline in every setting [arxiv:1904.07595 fig. 5]. The same comparison, with a HOG-distance and
logistic-regression detector, also flags adversarial inputs [arxiv:1904.07595 §3.3]. The method needs
a generator trained on the target domain, and the authors name false positives as the main open issue
[arxiv:1904.07595 §5]. Its released code is GPL-3.0-or-later. Card: [Detecting the Unexpected via Image
Resynthesis](/papers/1904.07595). See also [anomaly detection](/concepts/anomaly-detection).

## When to use which

All numbers are author-reported. mIoU (%) on the val split unless stated otherwise.

| Situation | Use | Why |
|---|---|---|
| Simplest strong baseline from an off-the-shelf ViT checkpoint | Segmenter, linear decoder | One linear layer on top of the encoder; Seg-L/16 reaches 52.25 on ADE20K multi-scale [arxiv:2105.05633 tab. 6] |
| Best accuracy on ADE20K or Pascal Context, compute not a concern | Segmenter mask transformer with ViT-L/16 | 53.63 ADE20K and 59.0 Pascal Context vs SETR-MLA's 50.28 and 55.8 in the same tables [arxiv:2105.05633 tab. 6] [arxiv:2105.05633 tab. 7] |
| Cityscapes (high-resolution street scenes) | SETR-PUP, T-Large | 82.15 multi-scale val; Segmenter's best is 81.3 on the same split [arxiv:2012.15840 tab. 6] [arxiv:2105.05633 tab. 8] |
| Throughput matters | Segmenter Base with patch 16 or 32 | Seg-B-Mask/16 runs at 76 images/s for 50.08 MS vs SETR-L MLA at 34 images/s for 50.28 [arxiv:2105.05633 tab. 6]; Seg-B-Mask/32 reaches about 400 images/s [arxiv:2105.05633 §4.3] |
| Need sharper boundaries / small objects and can pay compute | Smaller patches (P = 8) | +1.48 mIoU over P = 16, with small-object mIoU 41.43 vs 39.57, but 13x lower throughput [arxiv:2105.05633 tab. 3] [arxiv:2105.05633 tab. 4] |
| Only ImageNet-1k weights available | DeiT-B backbone | SETR-PUP-DeiT (T-Base) 79.45 on Cityscapes beats the ViT-21k T-Large SETR-PUP at 79.34, but trails it on ADE20K (46.34 vs 48.58 SS) [arxiv:2012.15840 tab. 2] [arxiv:2012.15840 tab. 4]; Seg-B†/16 47.08 vs Seg-B/16 48.06 on ADE20K [arxiv:2105.05633 tab. 3] |
| No pre-trained weights | Do not use these models | SETR-PUP from random init: 42.27 vs 78.39 on Cityscapes [arxiv:2012.15840 tab. 2]; Seg-S/16 from scratch: 4.42 to 12.51 vs 45.37 on ADE20K [arxiv:2105.05633 tab. 9] |
| Small training set | Expect a drop | Seg-S/16 falls from 45.37 (20k images) to 41.87 (8k) and 38.31 (4k) on ADE20K [arxiv:2105.05633 tab. 5] |
| Few labeled images, many unlabeled ones, CNN backbone is fine | U2PL with CutMix | 67.98 vs 45.77 supervised-only on VOC at 1/16 [arxiv:2203.03884 tab. 1]; needs a teacher and a memory bank of negatives per class [arxiv:2203.03884 §3.3] |
| Cityscapes with few labels | U2PL with AEL, not CutMix | CutMix variant is below AEL at 1/16, 1/8 and 1/4; only the AEL variant beats AEL at every fraction [arxiv:2203.03884 tab. 3] |
| Many labels (half or more) | Expect small gains | Blender split at 1/2: 80.50 vs 80.29 for AEL [arxiv:2203.03884 tab. 2] |
| Synthetic labels, real target images, no target labels, can afford heavy training | DISE | 45.4 vs 42.4 for output-space adaptation on GTA5 to Cityscapes; 88 GPU hours [arxiv:1903.12212 tab. 2] [arxiv:1903.12212 §3.3] |
| Same setting, simpler change to an existing adaptation pipeline | Maximum squares loss (with image-wise weights) | Swaps the entropy term for a squared-probability term; 46.4 vs 42.2 for the authors' entropy run [arxiv:1909.13589 tab. 2]; the paper states no extra discriminator is needed [arxiv:1909.13589 §1] |
| Classes arrive in steps and old images cannot be stored | PLOP | 54.64 vs 29.70 for MiB on VOC 15-1 [arxiv:2011.11390 §4.2] |
| Only image-level labels, need pseudo masks for training | ReCAM on CAM seeds | 54.8 vs 48.8 seeds on VOC; 70.9 vs 66.3 with IRN; cost roughly doubles on ResNet-50 seeds [arxiv:2203.00962 tab. 2] |
| Need to segment arbitrary text-named classes at test time | OpenSeg with EfficientNet-B7 | 72.2 PAS-20 zero-shot, but 28.6 on ADE-150 with prompt engineering [arxiv:2112.12143 tab. 2] |
| Connectomics electron-microscopy membranes, 2D tiles, small labelled set | FusionNet | Beats U-Net on Drosophila and zebrafish test sets, with no official code and no stated optimiser [arxiv:1612.05360 tab. 2] [arxiv:1612.05360 §3.3] |
| Large micrographs with few labelled images, one label per object, objects must keep their texture | Pixel-wise FCN with per-object majority vote (MVFCNN) | 93.94% object accuracy vs 66.50% for fine-tuned VGG16 on cropped objects, on one 21-image split [arxiv:1706.06480 tab. 1]; the headline denominator does not reconcile (see Contradictions) |
| Volumetric MRI or CT, boundary accuracy in 3D matters, compute is affordable | nnFormer, with an nnUNet ensemble if DSC matters most | Lower HD95 than nnUNet on three tasks, DSC close; the ensemble gives 87.51 DSC on Synapse [arxiv:2109.03201 tab. 5] |
| Real-time polyp segmentation, speed matters more than peak Dice | ColonSegNet | 182.38 FPS at Dice 0.8206 vs 35.00 FPS at 0.8757 for UNet-ResNet34 [arxiv:2011.07631 tab. V] |
| Six-camera vehicle or road map, real-time, small model | Cross-view Transformer | 36.0 IoU with 5M parameters and 35 FPS [arxiv:2205.02833 tab. 1] |
| Dense 3D semantic occupancy from LiDAR for a ground and air robot, SemanticKITTI-style labels | OccMamba | 59.9 IoU on the hidden test set at 22.1 FPS [arxiv:2408.10618 tab. 2]; the repository has no LICENSE file and the planner is not released, so reuse needs a re-implementation |
| Vector lane and crossing elements for a mapping pipeline, LiDAR available | HDMapNet (fusion) | 44.5 IoU and 30.6 mAP vs 32.9 and 22.7 for cameras only [arxiv:2107.06307 tab. I] [arxiv:2107.06307 tab. II] |
| Satellite image time series, per-pixel crop type, one study area | Bidirectional convolutional recurrent encoder | 89.7% overall accuracy on 17 classes in 2016 [arxiv:1802.02080 tab. 2]; check cross-year transfer before use |
| Compressing a segmenter into a small student | DIST distillation | 77.10 vs 74.21 student on Cityscapes with DeepLabV3-R18 [arxiv:2205.10536 tab. 7] |
| Map of reported remote-sensing numbers, not a method | Transformers in Remote Sensing survey | Lists benchmark numbers from original papers; no re-runs [arxiv:2209.01206 §5] |
| No labels at all, pixel classes wanted, a DINO ViT-B checkpoint is available | STEGO | 28.2 unsupervised mIoU on CocoStuff-27 vs 14.4 for PiCIE + H; the 27-class ontology is fixed, so the number is not comparable with supervised mIoU [arxiv:2203.08414 §4.2] |
| Medical image with faint targets, many network evaluations per image affordable | MedSegDiff++ | 87.5, 90.5 and 86.6 Dice vs 84.9, 88.2 and 84.2 for nnUNet [arxiv:2211.00611 tab. 1]; 25 samples x 100 steps per image [arxiv:2211.00611 §3.2] |
| Land-cover maps from weak labels, with a diffusion prior trained on similar patches | Plug-and-play diffusion with the land-cover constraint | Durham: 79.8% accuracy vs 78.9% for supervised weak-label training, but 46.4 vs 47.9 IoU [arxiv:2206.09012 tab. 1] |
| Synthetic street scenes with per-pixel, instance and depth labels, as pre-training before real data | Synscapes | 78.85 vs 76.56 for Cityscapes-only DeepLab v3+ after fine-tuning [arxiv:1810.08705 tab. 4]; dataset licence not stated in the paper [arxiv:1810.08705 §3] |
| Unknown objects on roads, no outlier training data, budget for a generator pass per frame | Image resynthesis with a discrepancy network | AUROC 0.79 to 0.93 on Lost and Found and a 60-image set [arxiv:1904.07595 fig. 5]; the released code is GPL-3.0-or-later |
| Aerial sensor shift, labelled source city, no target labels | GAN restyling of source patches, then fine-tuning | 0.35 to 0.52 overall accuracy, Potsdam to Vaihingen [arxiv:1905.03198 tab. 3]; the best epoch is chosen on target test data |
| Frozen per-pixel features for dense tasks, no labels | VADeR | 56.7 VOC mIoU vs 54.4 supervised, but 44.3 vs 47.1 on Cityscapes [arxiv:2011.05499 tab. 1]; no official code found |
| Polyp segmentation, speed on one GPU matters | HarDNet-MSEG | 86.7 FPS at 0.904 mean Dice on Kvasir-SEG [arxiv:2101.07172 tab. 1]; check the checkpoint rule before comparing |
| Building footprints from satellite imagery at continental scale, unlabelled imagery available | U-Net with edge weights, mixup and self-training | Self-training +0.057 mAP on average [arxiv:2107.12283 §11]; 516M polygons after deduplication [arxiv:2107.12283 §12] |
| New object class from a few labelled support images, GPU memory for the full correlation volume | VAT cost aggregation | 1-shot PASCAL-5i mIoU 67.9 vs 66.2 for HSNet, ResNet-101 [arxiv:2207.10866 tab. 1]; released configs use augmentation the paper does not |
| New medical task with a few labelled examples at hand, no training run | UniverSeg | Mean Dice 71.8 on six unseen datasets vs 50.1 for SE-net [arxiv:2304.06131 tab. 1]; 84.4 for per-task nnU-Net; training code not released |
| Promptable 3D medical volume, one click | SAM-Med3D | 76.27% Dice with one 3D point vs 36.83% for SAM-Med2D on each slice [arxiv:2310.15161 tab. 3] |
| Semantic map and camera tracking in one RGB-D model, real-time rendering | GS3LAM | Replica mIoU 96.63 over four scenes [arxiv:2603.27781 tab. 4]; the rasterizer's licence excludes commercial use |
| Small organ in 3D abdominal CT, want attention without a localisation cascade, small compute overhead acceptable | Attention U-Net (3D) | Pancreas DSC 0.840 vs 0.814 for U-Net, CT-150 with 120/30 split [arxiv:1804.03999 tab. 1]; 6.40 M vs 5.88 M parameters, 0.179 s vs 0.167 s per forward pass [arxiv:1804.03999 tab. 1] |
| Medical CT or MRI, global context and sharp boundaries needed, ImageNet ViT checkpoints available | TransUNet (R50-ViT-B/16, CUP decoder) | 77.48 Synapse DSC vs 75.57 for R50-AttnUNet [arxiv:2102.04306 tab. 1]; 89.71 ACDC vs 87.55 for R50-U-Net [arxiv:2102.04306 tab. 5]; the loss and checkpoint rule are not in the paper [arxiv:2102.04306 §4.2] |
| Binary road masks from aerial photos, small model, one dataset acceptable | Deep ResUnet | Relaxed break-even 0.9187 vs 0.9053 for U-Net on Massachusetts roads [arxiv:1711.10684 tab. II]; no seeds reported and the metric is not mIoU |
| Single-frame infrared frames, targets a few pixels wide, one centroid per target needed | DNA-Net-ResNet-18 | NUDT-SIRST Pd 98.73 and Fa 4.223 vs Pd 96.51 and Fa 9.261 for ALCNet [arxiv:2106.00487 tab. II]; baselines retrained by the authors |
| Indoor embodied-agent simulation with textured scenes and semantic labels, research use only | Replica (dataset, not a method) | 18 scenes, 88 classes, no learning benchmark [arxiv:1906.05797 tab. 1] [arxiv:1906.05797 §4]; scene terms are non-commercial |
| Cardiac MR or ultrasound, anatomically implausible masks are the problem, no extra cost at inference | ACNN (shape term during training only) | Endocardial Dice 0.939 vs 0.923 for a 3D baseline; Hausdorff gains not significant [arxiv:1705.08302 tab. 1]; the autoencoder schedule is not stated [arxiv:1705.08302 §2.2] |
| Small aerial building dataset (about 150 training images), ImageNet weights available | TernausNet (VGG11 encoder) | Validation IoU 0.686 vs 0.593 with random init, on 30 validation images, one run per setting [arxiv:1801.05746 §3] |
| Ten to forty labelled images, a diffusion model trained on the target domain, feature extraction affordable | DDPM features with an MLP ensemble | Mean IoU 49.4 on Bedroom-28 with 40 images, 38.2 with 10 [arxiv:2112.03126 tab. 2] [arxiv:2112.03126 tab. 4]; features cost about 2.2 GB per image |
| Only image-caption pairs, no masks, zero-shot label names at test time | GroupViT | Zero-shot 52.3 VOC 2012 and 22.4 PASCAL Context mIoU [arxiv:2202.11094 tab. 5]; background classes score near 1.5 IoU on Context [arxiv:2202.11094 app. C.3] |
| Mobile backbone with an SE block already in place, segmentation head on top, small parameter budget | Coordinate attention in place of SE | Plain-backbone, SE and CA on VOC val at stride 16: 70.84, 71.69, 73.32 [arxiv:2103.02907 tab. 8]; adds 0.06 M parameters over SE on ImageNet [arxiv:2103.02907 tab. 1] |
| Dense prediction with a plain pyramid transformer, ADE20K with Semantic FPN, 512 x 512 input | PVT v2-B2 (B2-Li is the cheaper variant) | 45.2 mIoU at 45.8 GFLOPs vs 39.8 for PVT v1-Small at 44.5 GFLOPs; B2-Li 45.1 at 41.0 GFLOPs [arxiv:2106.13797 tab. 5] |
| Unevenly spaced lidar or depth points, outputs needed at arbitrary positions | PCCN (continuous convolution) | 58.27 indoor mIoU vs 48.92 for SEGCloud [arxiv:2101.06742 tab. 1]; driving data in-house and no official code [arxiv:2101.06742 §4.2] |
| One camera feed, vehicles, drivable area and lane lines needed in real time on embedded hardware | YOLOP | 76.5 vehicle mAP50 at 41 FPS on a TITAN XP [arxiv:2108.11250 tab. 1]; ENet is faster on lanes, 100 FPS [arxiv:2108.11250 tab. 3]; code losses differ from the paper, see its card |
| LiDAR SLAM on roads with moving cars, labelled data and a segmenter available | SuMa++ | 0.70% vs 0.83% average translation error on KITTI training sequences, which are in-sample for the segmenter [arxiv:2105.11320 tab. II] |
| Open-vocabulary segmentation with few trainable parameters, CLIP kept frozen | SAN with ViT-B/16 | 27.5 ADE-150 mIoU with 8.4 M trainable parameters at 15.2 FPS [arxiv:2302.12242 tab. 2] [arxiv:2302.12242 tab. 3] |
| Judging whether a new 3D medical architecture beats the U-Net | nnU-Net with its residual-encoder presets as baseline | MedNeXt L and ResEnc L/XL are the strongest CNN entries; transformers trail the original nnU-Net [arxiv:2404.09556 tab. 1] |
| One labelled synthetic street-scene source, targets with other weather, lighting or cities, no target images | RobustNet (ISW loss, training only) | GTAV-trained ResNet-50: 36.58, 35.20 and 40.33 on Cityscapes, BDD-100K and Mapillary vs 28.95, 25.14 and 28.18 baseline [arxiv:2103.15597 tab. 1]; inference time unchanged [arxiv:2103.15597 tab. 6]; the Mapillary claim in the text does not match the Cityscapes table (see Contradictions) |
| Polyp segmentation with colour shift between collection sites | SANet | mean Dice 0.753 on ColonDB and 0.750 on ETIS vs 0.712 and 0.628 for PraNet [arxiv:2108.00882 tab. 1]; uses PraNet's split, so not comparable with HarDNet-MSEG's 880/120 numbers |
| No pretrained backbone wanted, many reverse chains per mask affordable | SegDiff | Cityscapes per-object mIoU 76.44 vs 73.70 for Spline-GCN [arxiv:2112.00390 tab. 1]; Dice 81.59 vs 79.55 for MedT on MoNuSeg [arxiv:2112.00390 tab. 3]; 30 chains in the paper, 9 in the code |
| Synapse-style abdominal CT, local detail and global context, ImageNet checkpoints for ResNet and Swin | HiFormer-B | Average Dice 80.39 and HD 14.70 vs 79.13 and 21.55 for Swin-Unet [arxiv:2207.08518 tab. 2]; the released forward pass does not pass the fused output to the decoder |
| Surgical video, binary tool masks needed, ImageNet-pretrained VGG encoder acceptable | TernausNet-16 | Binary IoU 83.60 vs 75.44 for U-Net [arxiv:1803.01207 tab. 1]; LinkNet-34 is faster (88 ms vs 184 ms) but has lower binary IoU, 82.36 [arxiv:1803.01207 tab. 1] |
| Skewed class frequencies, pretrained segmenter whose classifier can be retrained on a frozen backbone | DisAlign second stage | FCN R-50 on ADE20K 38.1 to 40.1 mIoU [arxiv:2103.16370 tab. 6]; the released stage-2 length and rho differ from the paper |
| Class names typed at test time, seen-class masks available, CLIP usable | zsseg (MaskFormer plus CLIP) | hIoU 37.8 vs 20.9 for FCN on COCO-Stuff [arxiv:2112.14757 tab. 2]; 77.5 vs 39.7 for CaGNet on Pascal VOC [arxiv:2112.14757 tab. 3] |
| Segmentation head with no learned class weights, prototype count should not grow with classes | ProtoSeg | +0.6 to +1.2 mIoU on ADE20K over four reproduced baselines [arxiv:2203.15102 tab. 1]; only a Cityscapes HRNet configuration is released |
| Single LiDAR sweep, distant objects poorly sampled, training compute for an auxiliary branch | JS3C-Net | 66.0 mIoU on SemanticKITTI vs 61.8 for the authors' SparseConv baseline [arxiv:2012.03762 tab. 1]; the completion branch is dropped at inference |
| Phone or edge device, latency measured on the device, ADE20K-style segmentation | EfficientFormerV2 (S2 or L) | ADE20K mIoU 42.4 for S2 and 45.2 for L; L reaches 83.3 top-1 at 2.7 ms [arxiv:2212.08059 tab. 2] [arxiv:2212.08059 tab. 3]; the search code is not in the repository |
| Pure ConvNet stack, no attention or deformable operators, large pre-training budget | ConvNeXt with UperNet | 46.7 mIoU for ConvNeXt-T against 45.8 for Swin-T [arxiv:2201.03545 tab. 4]; training memory 7.7 GB for ConvNeXt-B against 9.1 GB for ViT-B at batch 32 [arxiv:2201.03545 tab. 2] |
| Dilated backbone too slow at high resolution, ImageNet-pretrained stride-8 backbone available | FastFCN (JPU head) | 37.56 FPS against 18.77 at 512 x 512 [arxiv:1903.11816 tab. 2]; 51.05 against 49.91 mIoU for dilated EncNet on PASCAL-Context [arxiv:1903.11816 tab. 1] |
| High-resolution streams on a small budget, trained from scratch on Cityscapes | Lite-HRNet-30 | 76.0 val mIoU against 73.8 for Lite-HRNet-18 [arxiv:2104.06403 tab. 8]; no ImageNet pretraining [arxiv:2104.06403 §4.4] |
| Linear token cost wanted, ADE20K with a Semantic FPN or UperNet head | XCiT-S12/8 with FPN, or S24/8 with UperNet for the longer budget | 44.2 and 48.1 mIoU [arxiv:2106.09681 tab. 6]; 80k against 160k iterations [arxiv:2106.09681 §4.3] |
| Pyramid attention with a cheap attention cost, ADE20K budget | P2T-Small with Semantic FPN | 46.7 mIoU against 48.7 for P2T-Base [arxiv:2106.12011 tab. 3]; 80k iterations at batch 16 [arxiv:2106.12011 §IV-B] |
| Local detail and global context at a fixed budget, classification-style pre-training | LITv2-B (HiLo attention) | 47.2 mIoU with Semantic FPN [arxiv:2205.13213 tab. 3]; 300 epochs on eight V100 GPUs for classification [arxiv:2205.13213 §5.1] |
| Very few labelled LiDAR scans, synchronised camera data available | LaserMix++ | 56.2 against 45.4 supervised mIoU at 1% labels on SemanticKITTI [arxiv:2405.05258 tab. 2]; camera calibration needed [arxiv:2405.05258 §4.1] |
| Large CNN backbone with deformable sampling, Mask2Former pipeline, ImageNet-22K budget | InternImage-H | 62.9 mIoU on ADE20K with Mask2Former [arxiv:2211.05778 tab. 5]; joint pre-training set not listed [arxiv:2211.05778 app. A] |
| Linear-time mixing in a hybrid stack, non-commercial terms acceptable | MambaVision-B | 49.1 mIoU on ADE20K with UperNet for Base [arxiv:2407.08083 tab. 3]; code and weights non-commercial |
| Text-named targets from a chat model, one mask per target | PixelLM | 39.2 gIoU on MUSE at 7B against 12.8 for LISA-7B [arxiv:2312.02228 tab. 1]; LISA higher on refCOCO val and testB [arxiv:2312.02228 tab. 3] |
| Offline 3D scene with text queries, Replica-style data, non-commercial terms acceptable | Feature 3DGS | mIoU 0.782 at 14.55 FPS with the speed-up on Replica [arxiv:2312.03203 tab. 2]; code licence non-commercial |

Regularisation for the ViT models: encoder dropout off, stochastic depth around 0.1. On Seg-S/16,
dropout 0.2 cuts ADE20K mIoU from 45.37 to 36.63 [arxiv:2105.05633 tab. 2].

## Implementation sketch

The ViT pipeline:

```text
Input: image (H0, W0, 3), K classes, patch size P, pre-trained ViT with L layers

1. Pad the image so H and W are multiples of P (Segmenter code pads bottom/right
   and crops the prediction back).
2. Patch-embed: P x P patches -> linear projection -> N = HW/P^2 tokens of width D.
3. Add learned position embeddings, bilinearly resized from the pre-training grid
   to (H/P, W/P).
4. Run L pre-norm transformer blocks; keep the outputs of every layer if the
   decoder needs intermediate features (SETR-MLA, auxiliary heads).
5. Decode the (H/P, W/P, D) map to (H/P', W/P', K) scores:
     SETR-Naive : 1x1 conv, BN, ReLU, 1x1 conv to K
     SETR-PUP   : 4 x [conv, 2x upsample]
     SETR-MLA   : fuse M evenly spaced layers top-down, upsample, concatenate
     Seg-Linear : linear D -> K per token
     Seg-Mask   : append K class embeddings, run M = 2 blocks,
                  masks = normalise(patches) . normalise(classes)^T
6. Bilinearly upsample to (H, W); crop padding.
7. Train: per-pixel cross-entropy (+ SETR auxiliary heads on intermediate layers).
   Infer: sliding window at the training crop size, optional multi-scale + flip,
   argmax over K.
```

Where the variants differ in detail:

- SETR's PUP channel widths, MLA's $M$ and stream order, and the auxiliary loss weight are only shown
  in a figure or not stated at all [arxiv:2012.15840 §3.3] [arxiv:2012.15840 §4.1].
- Segmenter's paper applies softmax then layer norm to the upsampled masks; its released code instead
  applies LayerNorm to the patch-level masks before upsampling and leaves softmax to the loss, and
  adds learned projections before the scalar product. The card recommends following the code
  [arxiv:2105.05633 §3.2].
- GroupViT: the paper's assignment has no scale factor, and the released code adds one, plus an
  epsilon in the normalisation of the segment update. Its background thresholds (0.9 and 0.5 in the
  paper) also differ from the config values, so the numbers depend on which rules you follow
  [arxiv:2202.11094 §3.1] [arxiv:2202.11094 §4.1]. The card lists the code-side details.
- ACNN: the shape term's weight (0.01 in the paper) is described as "chosen experimentally" for the
  authors' intensity normalisation, so the value may not carry over to other scalings
  [arxiv:1705.08302 §2.2].
- Batch size on ADE20K: SETR uses 16, Segmenter 8; both use LR 1e-3 there and 1e-2 on Cityscapes
  [arxiv:2012.15840 §4.1] [arxiv:2105.05633 §4.2].
- Cost: global attention over $N$ tokens in every layer. A 768x768 Cityscapes crop at $P = 16$ is
  2,304 tokens per layer; SETR runs at stride 16 rather than the stride 8 of its dilated FCN baseline
  for GPU memory reasons [arxiv:2012.15840 §4.1], and Segmenter drops its Cityscapes Large mask
  decoder to one layer for memory [arxiv:2105.05633 §4.4].
- STEGO: the released config's defaults are a ViT-Small run, not the paper's ViT-B setting, and the
  paper's batch size is 32 where the config uses 16. Start from the paper's CocoStuff or Cityscapes
  row of loss weights and shifts [arxiv:2203.08414 app. Implementation Details]. The paper states the
  loss as sums, while the code averages each term, which changes the loss scale and so its interaction with the learning rate
  and loss weights [arxiv:2203.08414 §3.2] [arxiv:2203.08414 eq. 4].

The U2PL training step, in the paper's terms. The per-class loop is the part most often
mis-implemented:

```pseudocode
# One U2PL training iteration  [alg. 1] [eq. 1]
# Input: labeled batch (x_l, y_l), unlabeled batch x_u, epoch t of T, K classes
# The teacher is an EMA of the student and receives no gradient

# Teacher class probabilities on unlabeled pixels  [§3.2]
p = softmax(teacher(x_u))
# Per-pixel entropy of those probabilities  [eq. 5]
H = entropy(p)
# Unreliable share shrinks linearly from 20% towards 0  [eq. 7]
alpha = 0.2 * (1 - t / T)
# Threshold so that the top alpha share is unreliable  [§3.2]
gamma = quantile(H, 1 - alpha)
# Argmax pseudo-labels for reliable pixels, ignore the rest  [eq. 6]
pseudo = where(H < gamma, argmax(p), IGNORE)
# Supervised cross-entropy on labeled pixels  [eq. 2]
L_s = cross_entropy(student(x_l), y_l)
# Rescale the unsupervised loss by the share of kept pixels  [eq. 8]
L_u = cross_entropy(student(x_u), pseudo) * (pixels / kept)  # [eq. 3]

for c in classes:
    # Anchors: labeled pixels of c, plus reliable pixels pseudo-labeled c,
    # both with teacher p_c above 0.3  [eq. 9-11]
    anchors = anchor_set(c, y_l, pseudo, p, delta=0.3)
    # Positive: mean teacher feature over the anchors  [eq. 12]
    positive = mean(teacher_feat[anchors])
    # Negatives for c: labeled pixels whose top-3 contains c and not ground
    # truth c; unlabeled unreliable pixels whose rank of c is in [3, 20)
    # [eq. 14-15]
    queue[c].push(negatives_for(c, y_l, p, H, gamma))
    # Sample M anchors, N negatives from the queue, InfoNCE with tau = 0.5  [eq. 4]
    L_c += infonce(sample(student_feat[anchors], M), positive,
                   sample(queue[c], N), tau=0.5)

# Total loss; lambda_c = 0.1 in the paper's table  [eq. 1] [app. tab. A1]
loss = L_s + L_u + lambda_c * L_c
```

## Contradictions

- **Cross-paper numbers agree.** Segmenter's tables quote SETR as 48.64 / 50.28 on ADE20K, 55.8 on
  Pascal Context and 82.2 on Cityscapes [arxiv:2105.05633 tab. 6] [arxiv:2105.05633 tab. 7]
  [arxiv:2105.05633 tab. 8], matching SETR's own 48.64 / 50.28, 55.83 and 82.15
  [arxiv:2012.15840 tab. 4] [arxiv:2012.15840 tab. 5] [arxiv:2012.15840 tab. 6].
- **Is the decoder or the encoder the key?** SETR's results suggest the decoder choice is
  dataset-specific (PUP for Cityscapes, MLA for ADE20K) [arxiv:2012.15840 §4.2]; Segmenter's ablations
  show a linear head already competitive and the encoder size and patch size as the larger levers
  [arxiv:2105.05633 tab. 3] [arxiv:2105.05633 tab. 4]. The two studies use different decoders, batch
  sizes and pre-trained weights, so they cannot be compared directly. U2PL uses a CNN backbone, other
  splits and label fractions, so its numbers do not compare with the ViT tables either.
- **Two UDA papers, one benchmark.** DISE reports 45.4 on GTA5 to Cityscapes and maximum squares
  reports 46.4 with its full method [arxiv:1903.12212 tab. 2] [arxiv:1909.13589 tab. 2]. Both
  compare with an output-space adaptation at 42.4 [arxiv:1903.12212 tab. 2] [arxiv:1909.13589 tab. 2],
  but the maximum-squares paper reproduces its entropy baseline itself (42.2), and DISE's paper does
  not state its loss weights while its released code differs from the text [arxiv:1903.12212 §3.2].
  The two headline gaps are therefore not verified against one protocol.
- **Inconsistencies inside each paper.** SETR reports SETR-PUP-DeiT on ADE20K as 46.24 in one table and
  46.34 in another, and labels DeiT rows T-Large in tab. 3 but T-Base elsewhere
  [arxiv:2012.15840 tab. 3] [arxiv:2012.15840 tab. 4]. Segmenter reports Seg-L-Mask/16 as 51.30 SS in
  tab. 4 and 51.82 SS in tab. 6, without explaining the gap [arxiv:2105.05633 tab. 4]
  [arxiv:2105.05633 tab. 6]. DISE gives Tsai et al. 42.4 in one table and 42.6 for the same method in
  another [arxiv:1903.12212 tab. 2] [arxiv:1903.12212 tab. 4]. The maximum-squares text claims a
  "significant" gain from the multi-level head, while its table shows +0.9 and +1.2
  [arxiv:1909.13589 tab. 4]; quote the table, not the prose. RELLIS-3D gives KPConv 19.07 mIoU in the
  text and 19.97 in the table [arxiv:2011.12954 tab. 2].
- **Mixed metrics for nnFormer.** Against nnUNet on Synapse, nnFormer has lower HD95 (10.63 vs
  10.78) but lower DSC (86.57 vs 86.99) [arxiv:2109.03201 tab. 5]. The paper's text calls the DSC
  differences on tumour and Synapse not significant [arxiv:2109.03201 §4.3]. Pick the metric that
  matches the task.
- **Benchmark protocol for ColonSegNet.** The paper describes its 120 evaluation images as
  "validation" in one place and as "test" in the figure captions; its code uses the same images for
  checkpoint selection and for testing [arxiv:2011.07631 §V-B], so the reported scores are not a
  clean held-out result.
- **Steel micrographs: headline numbers.** The 93.94% headline has no stated denominator. §5.1 gives
  2262 test objects, but the confusion matrix covers 2205 classified objects plus 48 unsegmented ones
  [arxiv:1706.06480 §5.1] [arxiv:1706.06480 tab. 4]. The text says augmentation adds 2 percentage
  points, while the table gives 2.97 [arxiv:1706.06480 §6] [arxiv:1706.06480 tab. 2]. The 48.89%
  baseline is "shown" on the same data, without saying whether it was re-run or copied
  [arxiv:1706.06480 §5.3].
- **Speed-up figure.** Cross-view Transformer states a 4.5x inference speed-up over FIERY
  [arxiv:2205.02833 tab. 1]; the table's 35 and 8 FPS give about 4.4x. Quote the FPS values.
- **MedSegDiff gains, text versus table.** The text gives a 2.3% Dice and 2.4% IoU gain over
  EnsemDiff on brain tumour. Table 1 gives 88.7 to 90.5 Dice and 80.9 to 82.8 IoU, which is 1.8 and
  1.9 points [arxiv:2211.00611 §3.3] [arxiv:2211.00611 tab. 1]. The text calls the FF-Parser gain
  "near 1%", while Table 2 gives 0.7 to 0.8 points [arxiv:2211.00611 §3.4] [arxiv:2211.00611 tab. 2].
  Quote the tables.
- **Few-shot augmentation and voting.** VAT's text says it uses no augmentation and divides the vote
  sum by K [arxiv:2207.10866 §4.1] [arxiv:2207.10866 §3.6]. Its released configs apply photometric
  augmentation, and its code normalises votes by the largest count in the image (see its card).
  Reported numbers depend on which rule was used, so state the choice when comparing.
- **Text versus table, within one paper.** GS3LAM calls its tracking competitive, but SplaTAM's
  Replica ATE of 0.36 cm is below its 0.37 cm [arxiv:2603.27781 tab. 3]. Its text gives a 0.039 LPIPS
  gap to the runner-up, where the table gives 0.030 against GS-SLAM [arxiv:2603.27781 §4.2]
  [arxiv:2603.27781 tab. 2]. UniverSeg's text gives Dice gains of 7.3 to 34.9 over the best baseline,
  while its per-dataset table gives 6.4 to 33.0 [arxiv:2304.06131 §5.2] [arxiv:2304.06131 app. E].
  Quote the tables.
- **Abstract versus body.** The building pipeline's abstract gives mixup +0.12 and self-training +0.06
  mAP. The body gives +0.057 for self-training, and the mixup effect appears only in a figure
  [arxiv:2107.12283 fig. 8]. Use the body numbers.
- **Aerial adaptation, claim versus table.** The aerial GAN paper says classes other than Building and
  Tree are unaffected, but its table shows Low vegetation falling from 0.38 to 0.27
  [arxiv:1905.03198 §6] [arxiv:1905.03198 tab. 4].
- **TransUNet baseline row.** The per-organ DSC values in the R50-AttnUNet row do not average to the
  printed 75.57; the aorta value of 55.92 looks like the outlier [arxiv:2102.04306 tab. 1]. Check
  that row before using it as a baseline.
- **DNA-Net internal numbers.** Table IV gives Fa 4.228 for DNA-Net-ResNet18 and 9.581 for ALCNet
  under 4 pixels, while Table II gives 4.223 and 9.261 [arxiv:2106.00487 tab. II]
  [arxiv:2106.00487 tab. IV]. The real-data mixing table lists Fa 4.223 for the NUAA row, where
  Table II gives 2.353 [arxiv:2106.00487 tab. VIII] [arxiv:2106.00487 tab. II]. The mixing table
  also counts 791 synthetic training images, while the released NUDT split has 663 training ids
  [arxiv:2106.00487 tab. VIII]. Quote Table II and state the centroid tolerance.
- **GroupViT multi-label gains, text versus table.** The text attributes a 13.1-point gain to hard
  assignment and a 2.6-point gain to soft assignment [arxiv:2202.11094 §4.2]. Table 1 shows the
  13.1-point gain without hard assignment (12.0 to 25.1) and the 2.6-point gain with it (36.7 to
  39.3) [arxiv:2202.11094 tab. 1]. Quote the table. The text also puts the 2-stage gain on CC12M+YFCC
  at about 7% [arxiv:2202.11094 §4.2], against +15.1 mask mIoU in the table [arxiv:2202.11094 tab. 3].
- **ACNN speed and significance, text versus table.** The text says the super-resolution model is
  faster "by a factor of 5", but the table gives 0.29 s against 0.06 s, about 4.8 times
  [arxiv:1705.08302 §3.5] [arxiv:1705.08302 tab. 3]. The text calls the ultrasound Dice gain not
  significant, while the table gives p about 0.05 [arxiv:1705.08302 §3.4] [arxiv:1705.08302 tab. 2].

- **RobustNet claim versus its table.** The text says ISW beats the other methods on BDD-100K, Mapillary
  and SYNTHIA when trained on Cityscapes [arxiv:2103.15597 §5.2.1]. In that table, ISW scores 58.64 on
  Mapillary, below IRW at 59.20 and IW at 58.90 [arxiv:2103.15597 tab. 2]. Quote the table.
- **HiFormer repeated number.** The ResNet-18 row of its backbone ablation and the no-fusion row of its
  fusion ablation both report Dice 77.15 and HD 16.88, and the paper does not explain the repeat
  [arxiv:2207.08518 tab. 6] [arxiv:2207.08518 tab. 7]. Confirm the setting before quoting either.
- **PVT v2 segmentation sentence.** The text compares "PVT-Large" with ResNeXt101-64x4d and reports
  48.7 mIoU [arxiv:2106.13797 §4.3]. The table gives 48.7 for PVT v2-B5 and 42.1 for PVT v1-Large
  [arxiv:2106.13797 tab. 5]. Quote the table.
- **Transformers against nnU-Net on brain tumours.** The nnFormer card reports DSC 82.02 against
  81.87 for nnUNet, and a lower HD95, on its brain-tumour task [arxiv:2109.03201 tab. 5]. The
  nnU-Net Revisited benchmark ranks nnFormer below the original nnU-Net on BTCV, LiTS, BraTS2021,
  KiTS2023 and AMOS2022, and above it only on ACDC (92.40 against 91.54) [arxiv:2404.09556 tab. 1].
  The cards do not show that the two brain-tumour tasks are the same, and the protocols differ, so
  this stays open. Quote both with their setup.
- **OpenSeg and SAN on ADE-150.** OpenSeg reports 28.6 on ADE-150 for EfficientNet-B7 with prompt
  engineering [arxiv:2112.12143 tab. 2]. SAN reports 27.5 with ViT-B/16 and 32.1 with ViT-L/14
  [arxiv:2302.12242 tab. 2]. The backbones and training data differ. SAN's table has an OpenSeg row,
  which its card does not copy, so the two numbers are not from one comparison and do not give a
  ranking.

- **FastFCN's own mIoU values.** The text gives 52.10 and 54.03 mIoU on the 59-class PASCAL-Context
  protocol, while the abstract gives 53.13 for ResNet-101 [arxiv:1903.11816 §4.3]. The card does not
  resolve which is the reported result, so quote the table with its setup.
- **PixelLM and LISA on different benchmarks.** PixelLM reports 39.2 gIoU against 12.8 for LISA-7B on
  MUSE [arxiv:2312.02228 tab. 1], while LISA is higher on refCOCO val and testB [arxiv:2312.02228
  tab. 3]. The two benchmarks measure different things, so the results do not give a ranking.

## Papers

- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](/papers/2012.15840):
  SETR, a plain-ViT encoder at fixed 1/16 resolution with Naive, PUP and MLA convolutional decoders.
- [Segmenter: Transformer for Semantic Segmentation](/papers/2105.05633): a fully transformer
  pipeline with a linear or class-token mask decoder, plus ablations of model size, patch size,
  regularisation, data size and pre-training.
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](/papers/2203.03884): U2PL, a
  teacher-student method that keeps unreliable pixels as negatives in per-class queues for a
  pixel-level contrastive loss, on a ResNet-101 DeepLabv3+.
- [FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics](/papers/1612.05360):
  a U-Net-style network with residual blocks and summation skips, for electron-microscopy neuron
  boundaries.
- [Advanced Steel Microstructural Classification by Deep Learning Methods](/papers/1706.06480): a
  pixel-wise FCN with per-object majority voting for steel micrographs, on 21 image pairs.
- [Be Your Own Prada: Fashion Synthesis with Structural Coherence](/papers/1710.07346): text-guided
  outfit generation that first generates a clothing segmentation map and then renders each region.
- [Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders](/papers/1802.02080):
  a bidirectional convolutional recurrent encoder over Sentinel-2 time series for per-pixel crop
  classes.
- [All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation](/papers/1903.12212):
  DISE, which segments from a domain-invariant structure code and adapts by swapping texture codes.
- [Domain Adaptation for Semantic Segmentation with Maximum Squares Loss](/papers/1909.13589): a
  squared-probability target loss with image-wise class weights and a low-level guidance head.
- [Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning](/papers/2011.07631):
  a benchmark on Kvasir-SEG plus ColonSegNet, a small fast encoder-decoder.
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](/papers/2011.11390):
  continual segmentation with local pooled-output distillation and entropy-thresholded pseudo-labels.
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](/papers/2011.12954): an off-road dataset with
  image and LiDAR labels and baseline results; non-commercial terms, see Datasets and benchmarks.
- [HDMapNet: An Online HD Map Construction and Evaluation Framework](/papers/2107.06307): vector map
  elements from cameras, LiDAR or both, in bird's-eye view.
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](/papers/2109.03201): a 3D U-shaped
  transformer with local and global attention and skip attention, for medical volumes.
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143): OpenSeg,
  mask proposals with caption grounding for zero-shot segmentation.
- [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](/papers/2203.00962): ReCAM,
  a second classifier layer that sharpens CAM seeds from image-level labels.
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](/papers/2205.02833): a
  six-camera transformer that lifts image features into a top-down map with cross-view attention.
- [Knowledge Distillation from A Stronger Teacher](/papers/2205.10536): DIST, a correlation-based
  distillation loss that also reports Cityscapes segmentation results.
- [Transformers in Remote Sensing: A Survey](/papers/2209.01206): a survey that collects reported
  transformer numbers for remote-sensing tasks, without re-running them.
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](/papers/2203.08414):
  STEGO, a label-free head on a frozen DINO ViT-B that clusters distilled feature correspondences.
- [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](/papers/2211.00611):
  a conditional diffusion model for medical masks, with dynamic conditional encoding and FF-Parser.
- [Diffusion models as plug-and-play priors](/papers/2206.09012): a fixed diffusion prior with a
  differentiable task constraint at sampling time, with a land-cover segmentation experiment.
- [Anomaly Detection in Autonomous Driving: A Survey](/papers/2204.07974): a survey of corner-case
  detection for driving across sensors, with no experiments of its own.
- [Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing](/papers/1810.08705): 25,000
  path-traced synthetic street scenes with per-pixel, instance and depth labels.
- [Detecting the Unexpected via Image Resynthesis](/papers/1904.07595): flags unknown road objects by
  repainting the scene from its label map and comparing.
- [Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images](/papers/1905.03198):
  GAN restyling of labelled aerial patches, then fine-tuning.
- [Unsupervised Learning of Dense Visual Representations](/papers/2011.05499): VADeR, per-pixel
  contrastive features learned from view correspondences.
- [HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS](/papers/2101.07172):
  a HarDNet68 encoder with a light RFB and dense-aggregation decoder for polyps.
- [Continental-Scale Building Detection from High Resolution Satellite Imagery](/papers/2107.12283):
  U-Net building masks with edge weights, mixup and self-training.
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](/papers/2207.10866):
  VAT, transformer aggregation of the query-support correlation volume.
- [UniverSeg: Universal Medical Image Segmentation](/papers/2304.06131): in-context medical
  segmentation from a support set, with no training for a new task.
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](/papers/2310.15161):
  a promptable 3D ViT for medical volumes, trained from scratch.
- [GS3LAM: Gaussian Semantic Splatting SLAM](/papers/2603.27781): a semantic Gaussian field with pose
  tracking and keyframe mapping for RGB-D SLAM.
- [Attention U-Net: Learning Where to Look for the Pancreas](/papers/1804.03999): additive attention
  gates on the skip connections of a 3D U-Net, replacing a localisation cascade.
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](/papers/2102.04306):
  a ResNet-50 stem with a ViT-B/16 encoder and a cascaded upsampler, for CT and MRI.
- [Road Extraction by Deep Residual U-Net](/papers/1711.10684): a residual U-Net with no cropping for
  road masks in aerial images, evaluated with a relaxed break-even point.
- [Dense Nested Attention Network for Infrared Small Target Detection](/papers/2106.00487): nested
  U-shaped sub-networks with channel and spatial attention, and centroid extraction for infrared targets.
- [The Replica Dataset: A Digital Replica of Indoor Spaces](/papers/1906.05797): 18 indoor 3D scenes
  with HDR textures and semantic labels; research-only scene terms.
- [ACNN](/papers/1705.08302): an autoencoder-code shape penalty for cardiac segmentation and
  super-resolution, used only during training.
- [TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation](/papers/1801.05746):
  a U-Net with an ImageNet-pretrained VGG11 encoder for binary building masks.
- [Label-Efficient Semantic Segmentation with Diffusion Models](/papers/2112.03126): per-pixel features
  from a pretrained DDPM with an ensemble of small MLPs, for few-label segmentation.
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094): grouping
  transformer trained on image-caption pairs, with zero-shot label matching.
- [A Concise Survey on Lane Topology Reasoning for HD Mapping](/papers/2504.01989): a survey of lane
  topology methods that includes an aerial route built on road-pixel segmentation; no code, no new method.
- [Coordinate Attention for Efficient Mobile Network Design](/papers/2103.02907): a channel-attention
  block that pools along height and width, tested as an add-on in DeepLabV3 on VOC and Cityscapes.
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](/papers/2106.13797): a pyramid
  transformer with pooled keys and values, overlapping patches and a depthwise convolution in the
  feed-forward block, tested with Semantic FPN on ADE20K.
- [Deep Parametric Continuous Convolutional Neural Networks](/papers/2101.06742): a continuous
  convolution whose kernel is an MLP over relative 3D offsets, for indoor and lidar point clouds.
- [SuMa++: Efficient LiDAR-based Semantic SLAM](/papers/2105.11320): RangeNet++ labels added to
  surfel LiDAR mapping, with penalties for moving-object surfels and semantic ICP weights.
- [YOLOP: You Only Look Once for Panoptic Driving Perception](/papers/2108.11250): one shared
  encoder with vehicle, drivable-area and lane-line heads, trained jointly.
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242): a light side
  ViT on a frozen CLIP, whose attention biases steer CLIP's last blocks to classify mask proposals.
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](/papers/2404.09556):
  a controlled benchmark of CNN, transformer and Mamba 3D segmenters on six public datasets.
- [OMEGA: Efficient Occlusion-Aware Navigation for Air-Ground Robot in Dynamic Environments via State Space Model](/papers/2408.10618):
  OccMamba, a LiDAR network that predicts 3D semantic occupancy with Mamba blocks in separate
  semantic and geometric branches, plus a planner that avoids building an ESDF map.
- [Deep Semantic Segmentation of Natural and Medical Images: A Review](/papers/1910.07655): a survey of
  deep segmentation, with a loss-function section and no experiments of its own.
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](/papers/2103.15597):
  a training-only loss on the style-sensitive feature covariances of a DeepLabV3+.
- [Shallow Attention Network for Polyp Segmentation](/papers/2108.00882): SANet, a Res2Net decoder with a
  shallow attention gate, colour exchange in training and logit correction at inference.
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](/papers/2112.00390): a conditional
  diffusion model for masks, trained from scratch, averaging several sampled chains.
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](/papers/2207.08518):
  CNN and Swin pyramids fused by class-token cross-attention, for 2D medical images.
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](/papers/1803.01207): U-Net,
  TernausNet and LinkNet for binary, part and instrument-type masks in surgical video.
- [JS3C-Net](/papers/2012.03762): a LiDAR segmenter trained with a scene-completion decoder that is dropped at inference.
- [A Review on Deep Learning in UAV Remote Sensing](/papers/2101.10861): a literature review of deep
  learning on UAV imagery, with the authors' tallies by network, task, sensor and context.
- [A Survey of Deep Learning Techniques for Weed Detection from Images](/papers/2103.01415): a survey of 70
  deep-learning weed papers and 19 public datasets.
- [Distribution Alignment: A Unified Framework for Long-tail Visual Recognition](/papers/2103.16370): a
  per-class scale and offset with a learned gate, trained with re-weighted cross-entropy; the segmentation
  variant is reported on ADE20K.
- [Rethinking Semantic Segmentation: A Prototype View](/papers/2203.15102): non-learnable cluster-centre
  prototypes per class, with nearest-prototype prediction.
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757):
  a MaskFormer proposer with frozen CLIP labelling of masked crops against text prompts.
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](/papers/2301.03505):
  a review of vision transformers across medical imaging tasks, with compiled segmentation tables.
- [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](/papers/1903.11816):
  replaces dilated backbone convolutions with a learned Joint Pyramid Upsampling head.
- [Lite-HRNet: A Lightweight High-Resolution Network](/papers/2104.06403): a shuffle-block high-resolution
  network with conditional channel weighting in place of 1x1 convolutions.
- [XCiT: Cross-Covariance Image Transformers](/papers/2106.09681): attention across feature channels,
  with cost linear in the number of tokens.
- [P2T: Pyramid Pooling Transformer for Scene Understanding](/papers/2106.12011): attention keys and values
  from a pyramid of pooled maps, with depthwise convolutions.
- [Omnidata](/papers/2110.04994): a rendering pipeline for 21 mid-level labels and a 14.6M-image starter set.
- [A ConvNet for the 2020s](/papers/2201.03545): ConvNeXt, a pure ConvNet modernised from ResNet-50.
- [Image Data Augmentation for Deep Learning: A Survey](/papers/2204.08610): a taxonomy of augmentation
  methods, with segmentation, classification and detection runs for three of its groups.
- [Fast Vision Transformers with HiLo Attention](/papers/2205.13213): splits attention heads between
  pooled global attention and windowed local attention; LITv2 is built on it.
- [EfficientFormer: Vision Transformers at MobileNet Speed](/papers/2206.01191): a pure transformer with
  convolutional early blocks, sized by an iPhone 12 latency table.
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](/papers/2211.05778):
  a CNN backbone built on DCNv3, a deformable convolution with shared projections and grouped offsets.
- [Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059): EfficientFormerV2,
  a hybrid transformer found by a joint size-speed search.
- [RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](/papers/2304.03198):
  a drop-in convolution with per-window softmax attention over each receptive field.
- [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228): a multimodal language
  model that emits one mask per named target through learned codebook tokens.
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203):
  distils LSeg or SAM features into 3D Gaussians for text and prompt queries.
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](/papers/2405.05258): LaserMix
  and LaserMix++, semi-supervised LiDAR segmentation with camera distillation.
- [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](/papers/2407.08083): a backbone that mixes
  Mamba and self-attention in its later stages.
- [InDRiVE](/papers/2512.18850): a driving agent whose input is a segmentation image; not a segmentation method.