Semantic segmentation

Overview

Semantic segmentation predicts one class per pixel. Most papers here score it by mean intersection over union (mIoU) or a close relative such as Dice or IoU; the steel micrograph paper reports pixel and object accuracy instead 1706.06480 tab. 5, and so does the land-cover diffusion paper 2206.09012 tab. 1. The ViT papers report mIoU on ADE20K, Pascal Context and Cityscapes 2012.15840 §4.1 2105.05633 §4.1, and the semi-supervised paper on PASCAL VOC 2012 and Cityscapes 2203.03884 §4.2. The long-standing recipe is a fully convolutional encoder that shrinks the image step by step, so that wide context only becomes available in deep, low-resolution layers, plus add-ons (dilated convolutions, pyramid pooling, attention modules on top of the backbone) to widen that context 2012.15840 §3.1 2105.05633 §2.

The lines of work below attack different problems. The first asks how to get wide context from a plain transformer, the second how to train with few labels or none, and the rest how to move a segmenter to a new domain, a new set of classes, or a new kind of label. Diffusion models are a newer route: they generate the mask itself, or use a generative prior to constrain it. Specialised domains (3D medical volumes, endoscopy, steel micrographs, map-view driving, satellite time series, building footprints, aerial road masks, infrared frames, surgical video) reuse the same core idea with domain-specific encoders and output heads. Three later settings change what the user supplies at test time instead of retraining: a few labelled support images (few-shot), a support set given as context (in-context), or a click (promptable).

Plain ViT encoders. Both ViT papers replace the convolutional encoder with a plain Vision Transformer and treat segmentation as sequence-to-sequence over image patches 2012.15840 §1 2105.05633 §1. What they share:

• Patch tokens at fixed resolution. The image is split into P \times P patches, each linearly projected and given a learned position embedding, giving N = HW/P^2 tokens. The token grid is never downsampled; every layer runs global self-attention over all N tokens 2012.15840 §3.2 2105.05633 §3.1.
• Pre-trained ViT/DeiT weights. The encoder is initialised from an ImageNet classifier and the stored position embeddings are interpolated in 2D to the fine-tuning grid 2012.15840 §4.1 2105.05633 §4.2. Without this, results collapse (see When to use which).
• A light decoder maps the H/P \times W/P feature map to per-class scores and upsamples bilinearly to full resolution; training uses per-pixel cross-entropy 2012.15840 §3.3 2105.05633 §3.2.
• An MMSegmentation-style recipe: SGD with weight decay 0, polynomial LR decay, random resize 0.5 to 2.0, random crop and horizontal flip, sliding-window inference, and multi-scale testing at scales 0.5 to 1.75 plus flip 2012.15840 §4.1 2105.05633 §4.2.

The ViT variants differ almost entirely in the decoder.

Semi-supervised training. U2PL keeps a convolutional network (ResNet-101 with DeepLabv3+) and attacks the labeling cost instead: it trains on a few labeled images and many unlabeled ones in a teacher-student setup 2203.03884 §3.1 2203.03884 §4.1. Its question is what to do with the pixels the teacher is unsure about.

Variants

SETR: ViT encoder with convolutional decoders

SETR runs a ViT encoder on 16x16 patches (T-Base: 12 layers, width 768; T-Large: 24 layers, width 1024) and attaches one of three decoders to the 1/16 feature map 2012.15840 §3.2 2012.15840 tab. 1 2012.15840 §3.3:

• Naive: two 1x1 convolutions (with sync BN and ReLU between them) down to the class count, then a single 16x bilinear upsample.
• PUP (progressive upsampling): four rounds of convolution plus 2x upsampling.
• MLA (multi-level aggregation): takes M evenly spaced encoder layers, reduces each with convolutions, fuses neighbouring streams top-down by addition, upsamples each stream 4x, concatenates and upsamples 4x again.

Auxiliary segmentation heads on intermediate layers are trained alongside the main head 2012.15840 §4.1. Author-reported, the best decoder depends on the dataset: PUP leads on Cityscapes (82.15 mIoU val, multi-scale) while MLA leads on ADE20K (50.28 val, multi-scale) and Pascal Context (55.83) 2012.15840 tab. 4 2012.15840 tab. 5 2012.15840 tab. 6. The large models carry 305 to 318M parameters, against 68.59M for the ResNet-101 FCN baseline 2012.15840 tab. 2. Card: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

The same paper also tests a hybrid that feeds a ResNet-50 feature map into the transformer plus Naive decoder; with DeiT weights it reaches 78.28 on Cityscapes val at 80k iterations, below SETR-PUP-DeiT's 79.45 2012.15840 §4.1 2012.15840 tab. 2.

Segmenter: ViT encoder with a linear or mask-transformer decoder

Segmenter keeps the decoder transformer-only 2105.05633 §3:

• Linear: one point-wise linear layer from token width to K classes per patch, then bilinear upsampling 2105.05633 §3.2.
• Mask transformer: K learnable class embeddings, one per class, are appended to the patch tokens and processed by M extra transformer layers (M = 2); each class mask is the scalar product of the L2-normalised patch outputs with that class's output embedding 2105.05633 §3.2 2105.05633 eq. 4 2105.05633 §4.3. This is the one-slot-per-class form of object queries.

The mask decoder adds 0.4 to 1.6 mIoU over the linear one on ADE20K val, single-scale (for example Seg-B†/16 47.10 to 48.70, Seg-L/16 50.71 to 51.30), though not in every object-size bucket: Seg-B-Mask/16 scores lower than linear Seg-B/16 on large objects (52.66 vs 53.28) 2105.05633 tab. 4. The paper explores encoder size from Tiny (6M) to Large (307M) and patch sizes 8, 16 and 32 2105.05633 tab. 1 2105.05633 tab. 3. Best author-reported results come from Seg-L-Mask/16: 53.63 mIoU on ADE20K val (multi-scale, trained at 640 crops), 59.0 on Pascal Context and 81.3 on Cityscapes val 2105.05633 tab. 6 2105.05633 tab. 7 2105.05633 tab. 8 2105.05633 §4.2. Card: Segmenter: Transformer for Semantic Segmentation.

Hybrid CNN stem with a ViT encoder (TransUNet)

TransUNet keeps a ResNet-50 stem. Each cell of its 14x14 feature map becomes one token for a 12-layer ViT-B encoder, so no 16x16 pixel patch is cut from the image 2102.04306 §3.2 2102.04306 §4.2. A cascaded upsampler decodes the tokens back to full resolution and receives skip connections from the CNN at 1/8, 1/4 and 1/2 of the input; three skips is the best setting the paper tests 2102.04306 §3.2 2102.04306 §4.4. On Synapse multi-organ CT (224 input, 18 training and 12 evaluation cases) it reports 77.48 average DSC, against 75.57 for R50-AttnUNet and 74.68 for R50-U-Net 2102.04306 tab. 1. The hybrid stem matters: with the same upsampler, R50-ViT reaches 71.29 against 67.86 for a pure ViT with the same decoder 2102.04306 tab. 1. On ACDC cardiac MRI it reports 89.71 average DSC, against 87.55 for R50-U-Net 2102.04306 tab. 5. A 512 input raises Synapse to 84.36 DSC, but the sequence is about five times longer and the paper calls the cost much larger 2102.04306 tab. 2 2102.04306 §4.4. The paper does not state its loss, its learning-rate schedule or its checkpoint rule, so reproduce from the card's notes rather than the text alone. Card: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation.

Two pyramids with class-token fusion (HiFormer)

HiFormer feeds a three-level CNN into a three-level Swin pyramid through 1x1 convolutional skips 2207.08518 §3.1.1 2207.08518 §3.1.2. Only the two outer levels are fused. Each gets a class token that attends to the other level's patch tokens, and the cost is linear in the token count because the class token is the only query 2207.08518 §3.1.3 2207.08518 eq. 3. The decoder then sums the two outer levels and upsamples to full resolution 2207.08518 §3.2.

Author-reported on Synapse (8 abdominal organs), HiFormer-B reaches average Dice 80.39 and HD 14.70, against 79.13 and 21.55 for Swin-Unet and 77.48 and 31.69 for TransUnet 2207.08518 tab. 2. Removing the fusion module drops Dice to 77.15 and raises HD to 16.88 2207.08518 tab. 6. The paper reports +3.24 Dice from this module 2207.08518 §5. The variants differ in CNN depth and fusion size: B uses a ResNet-50 and 25.51 M parameters, against 23.25 M for S and 29.52 M for L 2207.08518 tab. 1 2207.08518 tab. 5. Two points need care before implementing. The paper does not say how the class-token output reaches the decoder, which reads only patch tokens 2207.08518 eq. 3 2207.08518 §3.2. The released forward pass also does not pass the fused output to the decoder, so it does not reproduce the Synapse gain as written (see its card). Card: HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation. See also hierarchical vision transformer.

Encoder scale and patch size

Within one decoder, the encoder dominates. On ADE20K val (linear decoder, single-scale), Segmenter goes from 39.03 (Ti/16) to 45.37 (S/16), 48.06 (B/16) and 50.71 (L/16); SETR likewise reports T-Large above T-Base 2105.05633 tab. 3 2012.15840 §4.2. Smaller patches help too: Seg-B/32 43.07, Seg-B/16 48.06, Seg-B/8 49.54, but throughput falls from 516 to 92 to 7 images/s on one V100 2105.05633 tab. 3. Since N = HW/P^2, halving P quadruples the token count 2105.05633 §3.1.

Pyramid transformer backbone with a Semantic FPN head (PVT v2)

PVT v2 is a four-stage pyramid transformer, so its features come at several resolutions and a decoder can fuse them 2106.13797 §3.5. Its attention pools keys and values to a fixed 7x7 grid, which makes the attention cost linear in the number of image tokens 2106.13797 §3.2 2106.13797 eq. 2. Queries still come from every token, so the projections still scale with the input 2106.13797 §3.2. Overlapping patch embedding and a 3x3 depthwise convolution in the feed-forward block are its other changes 2106.13797 §3.3 2106.13797 §3.4.

For ADE20K the segmentation head is Semantic FPN, trained with AdamW at learning rate 1e-4 for 40k iterations at batch 16, with polynomial decay and 512 x 512 crops 2106.13797 §4.3. Author-reported mIoU is 45.2 for PVT v2-B2, at 29.1 M parameters and 45.8 GFLOPs at 512 x 512, against 39.8 for PVT v1-Small (28.2 M, 44.5 GFLOPs) and 36.7 for ResNet-50 (28.5 M, 45.6 GFLOPs) 2106.13797 tab. 5. The linear-attention variant B2-Li reaches 45.1 at 26.3 M and 41.0 GFLOPs, and B5 reaches 48.7 2106.13797 tab. 5. The repository's segmentation configs cover PVT v1 only fpn_pvt_s_ade20k_40k.py, so the v2 segmentation setup has to be built from the paper's text. Card: PVT v2: Improved Baselines with Pyramid Vision Transformer. See also vision transformer.

Pyramid pooling attention for dense features (P2T)

P2T builds attention keys and values from a pyramid of average-pooled feature maps at four scales. Each pooled map passes through a shared 3x3 depthwise convolution, and the results are concatenated into a fixed number of tokens 2106.12011 §III-B.1 2106.12011 eq. 2-4. Queries still come from the full input, and the feed-forward block is an inverted bottleneck with a depthwise convolution 2106.12011 §III-B.2 2106.12011 eq. 10. With Semantic FPN on ADE20K, author-reported mIoU is 46.7 for P2T-Small and 48.7 for P2T-Base 2106.12011 tab. 3. Segmentation training uses AdamW at learning rate 1e-4 for 80k iterations at batch 16 with 512 x 512 crops 2106.12011 §IV-B. Fixed pooled sizes save 2.9 to 3.3 GB of memory [arxiv:2106.12011 tab. 8]. The pooling rounding and the head count are not stated 2106.12011 §III-B.1. Card: P2T: Pyramid Pooling Transformer for Scene Understanding. See also vision transformer.

Channel-covariance attention backbone (XCiT)

XCiT replaces token-to-token attention with attention across feature channels. Queries and keys are l2-normalised over the token axis, and a channel-by-channel cross-covariance matrix is softmaxed 2106.09681 §3.2 2106.09681 eq. 1. The cost grows linearly with the number of tokens, and every layer keeps the same token count 2106.09681 §3.3. A depthwise convolution after the attention adds local patch interaction before the feed-forward block 2106.09681 fig. 1. Author-reported ADE20K mIoU (single scale) is 44.2 for XCiT-S12/8 with a Semantic FPN head and 48.1 for XCiT-S24/8 with UperNet 2106.09681 tab. 6. Segmentation training uses AdamW at learning rate 6e-5 and batch 16, for 80k iterations with FPN and 160k with UperNet 2106.09681 §4.3. The softmax orientation in eq. 1 is ambiguous and differs from the released code, which the card recommends 2106.09681 app. C. Without the l2 normalisation, S12/16 fails [arxiv:2106.09681 tab. 4]. Card: XCiT: Cross-Covariance Image Transformers.

High-resolution backbone with cheap channel weighting (Lite-HRNet)

Lite-HRNet keeps the parallel high-resolution branches of HRNet but builds them from shuffle blocks. It replaces the costly 1x1 convolutions with conditional channel weighting: pooled features from every branch set per-channel weights for the others 2104.06403 §3.2 2104.06403 eq. 2 2104.06403 eq. 4. A Lite module is two weighting blocks and one fusion block [arxiv:2104.06403 tab. 1]. For segmentation the models are trained from scratch, without ImageNet pretraining, using SGD with polynomial learning rate 1e-2 for 160k iterations at batch 16 2104.06403 §4.4. The Cityscapes validation mIoU is 73.8 for the 18-channel model and 76.0 for the 30 model [arxiv:2104.06403 tab. 8]. The paper does not state the channel-reduction ratios, and the released repository has no segmentation code, so table 8 cannot be reproduced from it 2104.06403 §4.4. Card: Lite-HRNet: A Lightweight High-Resolution Network.

Learned upsampling instead of dilated backbones (FastFCN)

FastFCN drops the dilation from a standard backbone and recovers wide context with a learned head, Joint Pyramid Upsampling (JPU). JPU takes the outputs of the last three stages, maps each to a common width, upsamples the two deeper maps to the shallowest size, concatenates, and applies four parallel separable convolutions with dilations 1, 2, 4 and 8 1903.11816 §3.2 1903.11816 §3.3.3. A context head (PSP, ASPP or Encoding) follows 1903.11816 §3.2. On PASCAL-Context with ResNet-50, Encoding with JPU reaches 51.05 mIoU, against 49.91 for dilated EncNet [arxiv:1903.11816 tab. 1]. At 512 x 512 on a Titan Xp it runs at 37.56 FPS, against 18.77 for the dilated version 1903.11816 tab. 2. On ADE20K with ResNet-101, dilated EncNet is slightly higher, 44.65 against 44.34 1903.11816 tab. 4. The "more than three times" reduction in computation is stated in the text without a FLOP or memory table 1903.11816 §1 1903.11816 §5. The text gives 52.10 and 54.03 mIoU for the 59-class protocol, while the abstract gives 53.13 for ResNet-101; see Contradictions. Card: FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

Pure ConvNet backbone modernised from ResNet (ConvNeXt)

ConvNeXt starts from a ResNet-50 and applies a sequence of changes borrowed from transformer training and design, ending in a block of a 7x7 depthwise convolution, LayerNorm, a 4x-wide inverted bottleneck with one GELU, and layer scale 2201.03545 §2.4 2201.03545 §2.5 [arxiv:2201.03545 §2.6] 2201.03545 fig. 4. The stem is a 4x4 stride-4 patchify convolution, and downsampling uses separate 2x2 stride-2 convolutions with LayerNorm 2201.03545 §2.2. With UperNet on ADE20K, multi-scale mIoU is 46.7 for ConvNeXt-T against 45.8 for Swin-T 2201.03545 tab. 4. Pre-training uses AdamW at learning rate 4e-3, batch 4096 and weight decay 0.05 [arxiv:2201.03545 tab. 5]. Depthwise convolutions are slower than dense ones at equal FLOPs 2201.03545 §4. The kernel size is inconsistent inside the paper: §2.5 says 7x7 and appendix C says 5x5 [arxiv:2201.03545 app. C]. Card: A ConvNet for the 2020s. See also convolutional neural network.

Local-global attention with pooled heads (HiLo, LITv2)

HiLo splits the heads of each attention layer into two groups. A fraction alpha of the heads (Lo-Fi) attends from full-resolution queries to keys and values average-pooled by an s x s window, and the rest (Hi-Fi) run self-attention inside non-overlapping s x s windows 2205.13213 §4.1 2205.13213 eq. 6. LITv2 uses HiLo in its last two stages 2205.13213 §4.2. With a Semantic FPN head on ADE20K, author-reported mIoU is 47.2 for LITv2-B 2205.13213 tab. 3. Classification training is 300 epochs on eight V100 GPUs at total batch 1,024 2205.13213 §5.1. Speed rankings change across A100, V100 and RTX 6000 GPUs 2205.13213 tab. 6. For 12 heads, alpha times the head count is not an integer, and the released code rounds it down [arxiv:2205.13213 §4.1]. Card: Fast Vision Transformers with HiLo Attention.

Latency-driven mobile backbones (EfficientFormer and EfficientFormerV2)

EfficientFormer is a pure transformer with convolution-style early blocks: two stride-2 3x3 convolutions as the stem, then pooling-mixer blocks without layer norm, and attention only in the last stage 2206.01191 eq. 3 2206.01191 eq. 4 2206.01191 §4.1. Its depth and width come from a slimming step that uses a per-block latency table measured on an iPhone 12, applied to a supernet trained with Gumbel-softmax sampling 2206.01191 §4.2 2206.01191 alg. 1. Author-reported ImageNet-1K top-1 is 83.3 at 7.0 ms for the largest variant, and ADE20K mIoU is 45.1 for it 2206.01191 tab. 1 2206.01191 tab. 2. Latency is per image through Core ML, averaged over 1000 runs, and the batch size is not stated 2206.01191 tab. 1. The learning rate is 1e-3 scaled by batch size over 1024 2206.01191 §5.1.

EfficientFormerV2 keeps the family but adds a depthwise convolution in every feed-forward block, attention only in the last two stages, and a joint size-speed search 2212.08059 §3.1 2212.08059 §4.2.1. The search score weights latency and size, and each search evaluation takes about four GPU-minutes 2212.08059 §4.2.3. Author-reported ADE20K mIoU is 45.2 for the L model and 42.4 for S2 2212.08059 tab. 3; ImageNet-1K top-1 is 83.3 at 2.7 ms for L 2212.08059 tab. 2. The paper states the learning rate per 1024 batch and 16 GPUs, but not the per-GPU batch size 2212.08059 §5.1. Cards: EfficientFormer: Vision Transformers at MobileNet Speed and Rethinking Vision Transformers for MobileNet Size and Speed.

Deformable-convolution CNN backbones (InternImage)

InternImage builds on DCNv3, a 3x3 deformable convolution that starts from DCNv2 and makes three changes. One value projection is shared across the sampling points, the channels are split into groups that each have their own offsets, and the weights over the sampling points of each group are normalised with a softmax 2211.05778 §3.1 2211.05778 eq. 2. Blocks look like transformer blocks, with layer norm, a feed-forward network and residual connections 2211.05778 §3.2 2211.05778 fig. 3. Author-reported ImageNet-1K top-1 is 83.5 for the T model, against 82.1 for ConvNeXt-T 2211.05778 tab. 2. With Mask2Former on ADE20K, the H model reaches 62.9 mIoU 2211.05778 tab. 5. The largest models are pre-trained on ImageNet-22K, and H uses a joint pre-training set and M3I 2211.05778 §4.1. The joint set is not listed in the paper 2211.05778 app. A. Sharing the projection saves 42.0% of the parameters and 84.2% of GPU memory at the H scale 2211.05778 §4.3. Card: InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions.

Mamba and state-space hybrid backbones (MambaVision)

MambaVision starts with two convolutional stages. In its last two stages, each block's layers are split in half: the first half use a Mamba mixer, which runs a selective scan on half of the channels and a convolutional gate on the other half, and the second half use self-attention 2407.08083 §3.2.2 2407.08083 eq. 7. With UperNet at 512 x 512 on ADE20K, author-reported mIoU is 46.0 for Tiny and 49.1 for Base 2407.08083 tab. 3. ImageNet-1K top-1 is 82.3 for Tiny, at 6,298 images per second on an A100 at batch 128 2407.08083 tab. 1. The paper's stage widths in its appendix table are twice those in the released code 2407.08083 tab. S.2. The scan kernel comes from an external package that the repository does not include. The text says MambaVision-B has 56% fewer GFLOPs than MaxViT-B, but table 1 gives about 36% 2407.08083 §5.1. The card records a non-commercial code licence and non-commercial weights, so reuse is not commercial-safe. Card: MambaVision: A Hybrid Mamba-Transformer Vision Backbone.

Receptive-field attention convolutions (RFAConv)

RFAConv is a drop-in replacement for a k x k convolution. Attention maps are computed per window instead of being shared by overlapping windows, and each window's k² positions get their own softmax weights 2304.03198 §3.1 2304.03198 eq. 2. The window values are expanded, weighted, reshaped into k x k blocks and passed through a stride-k convolution 2304.03198 §3.3 2304.03198 eq. 3. RFCAConv builds on coordinate attention in the same way 2304.03198 §3.3. Author-reported ImageNet-1k top-1 for ResNet-18 rises from 69.59 to 71.23 with RFAConv 2304.03198 tab. 3. On PASCAL VOC 2012 with DeepLabV3+ at stride 16, mIoU is 64.6 for the baseline and 65.4 with RFAConv, and 68.0 with RFCAConv 2304.03198 tab. 10. The expanded tensor is k² times the size of the input, which the authors say limited-storage devices may not afford 2304.03198 §4.6. The paper describes the Unfold step as parameterless, while the released code adds a learned 1x1 grouped convolution and batch norm. Gains are uneven; coordinate attention does better on segmentation 2304.03198 §4.6. Card: RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks. See also channel attention.

Residual encoder-decoders with summation skips (FusionNet)

FusionNet is a U-Net-style network for neuron boundaries in electron-microscopy stacks. Inside every level it places residual blocks, and on the long skip connections it adds the encoder map to the decoder map instead of concatenating them 1612.05360 §3.1 1612.05360 tab. 1. It takes 640x640 tiles, uses eight-fold orientation enrichment with elastic warps and noise, and averages the eight predictions at test time 1612.05360 §3.2. Author-reported, it beats the original U-Net on the Drosophila test set (V_rand 0.97804 vs 0.97276) and on a zebrafish test volume (Dice 0.9461 vs 0.9085) 1612.05360 tab. 2 1612.05360 tab. 3. The optimiser, learning rate, batch size and epoch count are not stated, so the training recipe must be chosen by the implementer 1612.05360 §3.3. Card: FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics.

Attention gates on skip connections (Attention U-Net)

Attention U-Net adds an additive attention gate to each skip connection of a 3D U-Net. The gate combines the skip features with a coarser gating signal and produces one sigmoid coefficient per voxel, or one per class; the skip features are then scaled by it 1804.03999 §2 1804.03999 eq. 1 1804.03999 eq. 2. The lowest skip connections are not gated, and each decoder scale gets an auxiliary output 1804.03999 §2. The stated aim is to replace the two-stage cascade, which first localises a small organ and then segments it, with one network 1804.03999 §1.

Author-reported on CT-150 with 120 training and 30 test scans, pancreas DSC is 0.840 for Attention U-Net against 0.814 for U-Net; the ± values are shown as printed, and the paper does not say what they measure 1804.03999 tab. 1 1804.03999 §3. On the public Pancreas-CT set the gain is smaller before fine-tuning, 0.712 against 0.690 1804.03999 tab. 3. The gates cost 6.40 M parameters against 5.88 M, and 0.179 s against 0.167 s per forward pass [arxiv:1804.03999 tab. 1]. The significance tests behind the p-values are not named 1804.03999 §3. The released code differs from the paper in its optimiser and in some layers, so check its card before reproducing 1804.03999 §3. Card: Attention U-Net: Learning Where to Look for the Pancreas. See also attention mechanism.

Position-aware channel attention in a DeepLabV3 model (coordinate attention)

Coordinate attention is a channel-attention block for mobile networks that keeps position. It pools along height and width separately and produces one gate per row and one per column 2103.02907 §3.2.1 2103.02907 §3.2.2. The segmentation test adds it to a DeepLabV3 model with a MobileNetV2 backbone; the ASPP branches use depthwise-separable 3x3 convolutions and weight decay is 4e-5 2103.02907 §4.4.2. See also channel attention.

Author-reported mIoU on Pascal VOC 2012 val is 73.32 at output stride 16 and 73.96 at output stride 8 with coordinate attention, against 70.84 and 71.82 for the plain backbone and 71.69 and 72.52 with SE 2103.02907 tab. 8. CBAM is 0.4 above the baseline at stride 16 and 0.15 below it at stride 8 2103.02907 tab. 8. On Cityscapes val at output stride 8 the paper reports 74.0, against 71.4 for the baseline, 72.2 with SE and 71.4 with CBAM 2103.02907 tab. 9. The block adds 0.06 M parameters over SE on ImageNet, with reduction ratios that differ from SE's 2103.02907 tab. 1. The repository holds the block and a MobileNetV2 model but no segmentation code README.md README.md README.md, so these numbers come from the paper's tables and were not re-run. Card: Coordinate Attention for Efficient Mobile Network Design.

Training-only shape prior on the output (ACNN)

ACNN adds a shape term to the segmentation loss and drops it at inference. A convolutional autoencoder, trained on label maps, maps the predicted and the true label maps to 64-dimensional codes, and the loss penalises the squared distance between the two codes 1705.08302 §2.2 1705.08302 eq. 1. For cardiac MR super-resolution, where the output is an intensity image, a separate predictor maps images into the same code space and is used only during training 1705.08302 §2.4.

Author-reported on cardiac MR, the endocardial Dice is 0.939 against 0.923 for a 3D network with sub-pixel up-sampling, and the myocardial Dice is 0.811 against 0.773 1705.08302 tab. 1. The Hausdorff differences are not significant in that table 1705.08302 tab. 1. On CETUS 3D ultrasound, the ejection-fraction correlation is 0.913 against 0.885 for an FCN [arxiv:1705.08302 tab. 2]. For super-resolution, SSIM is 0.796 against 0.784 for a 3D U-Net, at 0.06 s per 120 x 120 x 12 input 1705.08302 tab. 3. The autoencoder's training schedule, the input the encoder receives and the loss reductions are not stated, so the shape term cannot be reproduced from the text alone 1705.08302 §2.2 1705.08302 §3.2. Card: ACNN. See also shape-prior regularization.

Residual U-Net for aerial road masks (Deep ResUnet)

Deep ResUnet replaces the plain convolution blocks of U-Net with full pre-activation residual units and drops the cropping before each concatenation, because input and output have the same size 1711.10684 §2.1.3 1711.10684 fig. 2. The text gives 15 convolution layers against 23 for U-Net, and about 7.8 M parameters against 30.6 M 1711.10684 §2.1.3 1711.10684 §3.4. Training uses an MSE loss with SGD on 30,000 random 224 x 224 patches, without augmentation; momentum and weight decay are not stated 1711.10684 §3.2 1711.10684 §2.2.

The metric is a relaxed break-even point with a 3-pixel slack, not IoU or mIoU. On the 49-image Massachusetts roads test set it reports 0.9187, against 0.9053 for U-Net and 0.9047 for Saito et al. 1711.10684 tab. II. The gap to U-Net is 0.013 points, and the paper reports no seeds or spread, so the difference is not established 1711.10684 tab. II. Only one dataset is tested 1711.10684 §3.1. The paper has no official code. Card: Road Extraction by Deep Residual U-Net.

ImageNet-pretrained VGG encoder in a U-Net (TernausNet)

TernausNet replaces the encoder of a U-Net with the convolutional part of VGG11, initialised from ImageNet weights, and segments buildings against background in aerial images 1801.05746 §2 1801.05746 fig. 1. Training minimises binary cross-entropy minus the log of a soft Jaccard term, and the probability map is thresholded at 0.3 1801.05746 eq. 4 1801.05746 §2. Adam with learning rate 0.001 runs for 100 epochs on random 768 x 768 crops 1801.05746 §3.

Author-reported validation IoU on the Inria aerial set is 0.593 with random (LeCun-uniform) initialisation, 0.686 with the ImageNet encoder, and 0.687 with a full network pretrained on Carvana 1801.05746 §3. These come from 30 validation images and one run per setting, with no spread reported, so the gain is one experiment rather than a broad result 1801.05746 §3. The text describes seven encoder convolutions, while the released code uses eight, and the skip connections and bottleneck differ in detail (see its card) 1801.05746 §2. Card: TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation. See also transfer learning.

Surgical instruments: U-Net-family encoders for tool masks

Three U-Net-style networks and one LinkNet segment robot instruments in surgical video frames, at three levels: a binary tool mask, three tool parts, and seven instrument types 1803.01207 §2.1 1803.01207 §3. U-Net is trained from scratch, while TernausNet-11 and TernausNet-16 use VGG encoders and LinkNet-34 uses a ResNet-34 encoder 1803.01207 §2.2. The loss is cross-entropy plus a soft Jaccard term, written H − log J 1803.01207 §2.3 1803.01207 eq. 3. The paper applies a 0.3 threshold to the binary output 1803.01207 §2.3.

Author-reported on the MICCAI 2017 challenge training data 1803.01207 §2.1: binary IoU is 83.60 for TernausNet-16 against 75.44 for U-Net; parts IoU is 65.50 for TernausNet-16 against 48.41 for U-Net; instrument-type IoU is 34.61 for TernausNet-11 1803.01207 tab. 1. LinkNet-34 is the fastest on the binary task, at 88 ms on a GTX 1080 Ti against 184 ms for TernausNet-16, but its parts IoU is 34.55 1803.01207 tab. 1. The paper attributes the weak multi-class results to few training examples for some instrument types 1803.01207 §3. It does not say which split its table uses, and the released evaluation differs from the paper (see its card), so treat the numbers as one study's results. Card: Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning. See also transfer learning.

Pixel-wise FCN with per-object voting for microscopy

MVFCNN labels steel micrographs with a fully convolutional network. The backbone is VGG16 with its fully connected layers turned into convolutions, plus up-sampling and skip layers [arxiv:1706.06480 §4.2] 1706.06480 §3.1. Large SEM images are cut into 1000x1000 px windows, every pixel is labelled, the labels are stitched back together, and each object takes the majority label of its pixels 1706.06480 §4.2. Objects are never resized, which the paper argues keeps the texture a classifier needs 1706.06480 §4.1 1706.06480 §6.

Training runs in three stages, FCN-32s, then FCN-16s, then FCN-8s, each starting from the weights of the previous one. The learning rates are 1e-10, 1e-11 and 3e-12, and the paper says larger rates made the loss explode 1706.06480 §4.3.3. Per-class sliding-window strides balance the patch counts across classes 1706.06480 §4.3.4. The exact per-class strides are not stated.

Author-reported on 21 SEM and LOM image pairs, with one split of 11 training and 10 test images 1706.06480 §5.1: 93.94% object accuracy on SEM, against 66.50% for a fine-tuned VGG16 on cropped objects and 48.89% for the earlier Pauly et al. method 1706.06480 tab. 1. Training from scratch gives 55.50%, fine-tuning 87.98%, class balancing 90.97% and augmentation 93.94% 1706.06480 tab. 2. On LOM images the same method reaches 70.14% 1706.06480 tab. 3. Bainite is the weakest class, with 37.32% pixel accuracy in the best setup 1706.06480 tab. 6. Card: Advanced Steel Microstructural Classification by Deep Learning Methods.

Map-view and 3D segmenters for driving and volumes

Four papers in this group move the segmenter's output space rather than its encoder.

• Vehicle and road maps from six cameras. The Cross-view Transformer encodes each camera with EfficientNet-B4, uses cross-view attention to lift the features into a 25x25 map embedding, and decodes it to 200x200 2205.02833 §3.2 2205.02833 §4. Each camera key carries a positional embedding built from its intrinsics and extrinsics, so no explicit depth or homography is computed 2205.02833 §3.1. On nuScenes Setting 2 it reaches vehicle IoU 36.0 with 5M parameters and 35 FPS, against 35.8 with 7M parameters and 8 FPS for static FIERY 2205.02833 tab. 1. The paper reports 37.5 against FIERY's 37.7 in Setting 1 2205.02833 tab. 1. Card: Cross-view Transformers for real-time Map-view Semantic Segmentation. See also BEV representation.
• Vector HD-map elements. HDMapNet predicts lane dividers, lane boundaries and pedestrian crossings in bird's-eye view. Camera features go through a learned per-camera MLP and are warped with extrinsics; LiDAR uses a PointPillars-style encoder; three heads output semantics, instance embeddings and lane direction 2107.06307 §3.1. Camera and LiDAR fusion reaches 44.5 IoU and 30.6 mAP, against 32.9 IoU and 22.7 mAP for cameras alone 2107.06307 tab. I 2107.06307 tab. II. Card: HDMapNet: An Online HD Map Construction and Evaluation Framework.
• Volumetric transformers. nnFormer is a 3D U-shaped network that interleaves convolution and self-attention, attends locally in the encoder and globally in the bottleneck, and uses skip attention instead of concatenation 2109.03201 §1 2109.03201 §3. Against an nnUNet rerun on the brain-tumour task it gives HD95 4.42 mm vs 4.60 and DSC 82.02 vs 81.87 2109.03201 tab. 5. Card: nnFormer: Interleaved Transformer for Volumetric Segmentation.
• LiDAR occupancy for an air-ground robot. OccMamba assigns a class to every cell of a 256 × 256 × 32 voxel grid built from a LiDAR point cloud. It has a semantic branch and a geometric branch, both with Mamba blocks, fused in bird's-eye view 2408.10618 §3.1. On the hidden SemanticKITTI test set it reports IoU 59.9, mIoU 25.0 and 22.1 FPS 2408.10618 tab. 2. On validation it reports IoU 58.6 and mIoU 25.2 with 23.8 M parameters, against 149.6 M for MonoScene 2408.10618 tab. 3. Removing the geometric or the semantic Mamba blocks gives validation mIoU 24.7 and 24.1 2408.10618 tab. 4. The text gives different gains from these tables, so use the tables 2408.10618 §5.2. Card: OMEGA: Efficient Occlusion-Aware Navigation for Air-Ground Robot in Dynamic Environments via State Space Model.

Point-cloud segmentation with continuous convolutions (PCCN)

Points from a lidar or depth sensor sit unevenly in space, so a grid convolution does not apply directly. PCCN defines the kernel as a small MLP over relative 3D offsets, which lets one layer take scattered inputs and produce outputs at any location 2101.06742 §3.1 [arxiv:2101.06742 §3.2]. Its locality window is the K nearest neighbours or a radius 2101.06742 §3.3. The kernel is factorised so that only one kernel value per output channel is stored for each point pair 2101.06742 §3.3.

Author-reported, the indoor model reaches 58.27 mIoU on the Stanford indoor set against 48.92 for SEGCloud and 41.09 for PointNet 2101.06742 tab. 1. On in-house driving scans, the combination of a trained 3D-FCN with seven residual PCCN layers reaches 58.06 mIoU, against 49.28 for 3D-FCN alone; the PCCN-only model is smaller (9.34 MB) but scores lower on mIoU, 46.35 2101.06742 tab. 2. The driving and flow data are in-house and not stated to be released, and no official code was found, so these numbers cannot be checked from public sources 2101.06742 §4.2. The paper's own dataset sizes disagree, so do not quote one as fact 2101.06742 §1 2101.06742 §4. Card: Deep Parametric Continuous Convolutional Neural Networks. See also continuous convolution.

Sparse LiDAR sweeps with a scene-completion teacher (JS3C-Net)

JS3C-Net labels the points of one LiDAR sweep. A sparse-voxel U-Net gives each point a class score and a shape embedding 2012.03762 §3.2. A second, scene-completion decoder predicts a dense voxel volume of the whole scene, and its targets come from voxels built from 70 merged sweeps 2012.03762 §3.3 2012.03762 §4.1. A point-voxel interaction module links the two branches. The completion branch is used in training only 2012.03762 §3.4 2012.03762 §3.6. The two losses are combined with learned uncertainty weights 2012.03762 §3.5.

Author-reported, the segmentation model reaches 66.0 mIoU on the SemanticKITTI benchmark, against 61.8 for the authors' own SparseConv baseline and 58.8 for KPConv 2012.03762 tab. 1. On SemanticPOSS it reaches 60.2 average IoU, against 57.2 and 55.2 2012.03762 tab. 2. On the validation split, dropping the joint learning lowers segmentation mIoU from 67.5 to 63.1 2012.03762 tab. 4. The latency table lists 471 ms for the segmentation backbone alone, and a second row with 107 ms and 0.4 M more parameters for the version with the auxiliary and interaction modules 2012.03762 tab. 5. The released code differs from the paper in its input feature (remission, not coordinates) and in feeding raw logits to the completion branch, so check the card before reproducing kitti_dataset.py train.py. Card: JS3C-Net. See also semantic scene completion.

Semi-supervised LiDAR mixing with camera distillation (LaserMix, LaserMix++)

LaserMix cuts each LiDAR scan into inclination bands and swaps alternate bands between two scans, together with their labels, to make new training scans 2405.05258 §3.2 2405.05258 eq. 7. In the semi-supervised setup, a teacher updated as a moving average of the student pseudo-labels unlabelled scans above a confidence threshold 2405.05258 §3.3. LaserMix++ adds image-feature distillation into the LiDAR branch and a CLIP-aligned loss 2405.05258 eq. 13 2405.05258 eq. 15, combined in 2405.05258 eq. 16. Author-reported mIoU on SemanticKITTI with Cylinder3D and 1% labels is 45.4 for the supervised baseline, 50.6 with LaserMix and 56.2 with LaserMix++ 2405.05258 tab. 2. Labelled fractions run from 1% to 50% on eight A100 GPUs 2405.05258 §5.1 2405.05258 §5.2. The method needs calibrated, synchronised camera and LiDAR data 2405.05258 §4.1. The camera-side parts of LaserMix++ have no released code, and the image model is not named 2405.05258 §4.4. Results are single runs with no variance 2405.05258 §5.4, and the LiM3D comparison uses test-time ensembles, so it is not like for like 2405.05258 §5.3. Card: Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving. See also semi-supervised learning.

One encoder, three driving heads (YOLOP)

YOLOP shares one encoder across three driving tasks: vehicle detection, drivable-area segmentation and lane-line segmentation 2108.11250 §3 2108.11250 §3.2. The encoder is CSPDarknet with SPP and FPN 2108.11250 §3.1. The detection head is a YOLOv4-style single-stage detector on three scales 2108.11250 §3.2.1. Both segmentation heads read the stride-8 FPN map and upsample to full resolution 2108.11250 §3.2.2. The paper trains end to end by default; a three-stage schedule is also tested, and Algorithm 1 moves to the next stage when the loss falls below a threshold 2108.11250 §3.4 2108.11250 alg. 1.

On BDD100K validation, author-reported, the single joint model reaches 89.2 recall and 76.5 mAP50 for vehicles, 91.5 mIoU for drivable area, and 70.50 lane accuracy with 26.20 IoU 2108.11250 tab. 1 2108.11250 tab. 2 2108.11250 tab. 3. It runs at 41 FPS on a TITAN XP 2108.11250 tab. 1. Table 5 compares the joint model with single-task models: the joint drivable-area mIoU is 91.5 against 92.0 single-task, and the joint lane accuracy is 70.5 against 79.6 for a lane-only model 2108.11250 tab. 5. The joint model's time is 24.4 ms; the three single-task times sum to 45.3 ms, a sum the card computes from that table 2108.11250 tab. 5. For lanes, ENet is much faster, at 100 FPS 2108.11250 tab. 3. Card: YOLOP: You Only Look Once for Panoptic Driving Perception. See also multi-task learning and object detection.

Narrower models: satellite time series, a survey and polyps

Several papers here are narrower in scope. A recurrent convolutional encoder turns Sentinel-2 time series into per-pixel crop labels for 17 classes, reaching 89.7% overall accuracy in 2016 and 89.5% in 2017 1802.02080 tab. 2 1802.02080 §5.2. The study trains and tests on both seasons, so it does not show transfer to an unseen year 1802.02080 §4 1802.02080 §5.2. Card: Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders. A survey, Transformers in Remote Sensing, collects reported segmentation numbers, for example 91.0 overall accuracy on Potsdam for Trans-CNN 2209.01206 tab. 4, but does not re-run any method 2209.01206 §5 2209.01206 §6.

For colonoscopy, a benchmark on Kvasir-SEG compares detectors and segmenters on speed and accuracy. The proposed ColonSegNet, a small residual encoder-decoder with squeeze-and-excitation blocks, reaches Dice 0.8206 at 182.38 FPS, while the best Dice in the table is UNet with ResNet34 at 0.8757 and 35.00 FPS 2011.07631 tab. V. Card: Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning.

HarDNet-MSEG targets the same polyp task with a HarDNet68 encoder and a light decoder. The decoder uses RFB blocks and dense aggregation, and drops the highest-resolution encoder level 2101.07172 §3.1 2101.07172 §3.2. Under the Kvasir-SEG 880/120 split it reports 0.904 mean Dice at 86.7 FPS 2101.07172 tab. 1. Under the PraNet protocol, trained on 1450 images from Kvasir-SEG and CVC-ClinicDB, it reports 0.912 2101.07172 tab. 2. The two setups differ in input size, optimiser and training data, so the figures are not one benchmark 2101.07172 §4.2. The card lists training details the paper leaves out, including the loss and the checkpoint rule, so check it before reproducing. Card: HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS.

SANet also targets polyps, with a Res2Net encoder and a decoder that gates shallow features with deep ones. In training it swaps colour statistics between images in LAB space, so the input keeps its content but takes another image's colours 2108.00882 §3.1 2108.00882 alg. 1. Its shallow attention gate multiplies early features by a ReLU of the upsampled deep features 2108.00882 §3.2 2108.00882 eq. 1. At inference, it divides the positive and the negative logits by the fraction of each in the image, which needs no retraining 2108.00882 §3.3 2108.00882 eq. 3.

Author-reported mean Dice on five polyp sets is 0.904 on Kvasir, 0.916 on CVC-ClinicDB, 0.753 on ColonDB and 0.750 on ETIS. PraNet scores 0.898, 0.899, 0.712 and 0.628 on the same sets 2108.00882 tab. 1. The paper reports about 72 FPS on an RTX 2080 Ti, against 64 for PraNet 2108.00882 §4.2. It follows PraNet's split 2108.00882 §4.1, which differs from the 880/120 Kvasir split behind HarDNet-MSEG's 0.904, so the two Kvasir numbers are not comparable. The released code has no shallow attention module, so build that gate from the equations rather than from the repository (see its card). Card: Shallow Attention Network for Polyp Segmentation.

Non-learnable prototypes as class heads (ProtoSeg)

ProtoSeg removes the learned class weights, or queries, from a segmentation head. Each class is described by K = 10 prototypes, which are centres of clusters of training-pixel embeddings 2203.15102 §3 2203.15102 §5.1. A pixel's class score is its best cosine similarity to any of that class's prototypes, and the pixel takes the highest-scoring class 2203.15102 eq. 5 2203.15102 eq. 6. The clusters come from online Sinkhorn clustering with an equal-size constraint, and the prototypes move by a momentum average instead of gradient descent 2203.15102 eq. 8 2203.15102 eq. 10 2203.15102 eq. 14 2203.15102 alg. 1. The training loss adds a pixel-prototype contrastive term and a compactness term to cross-entropy 2203.15102 eq. 11 2203.15102 eq. 12 2203.15102 eq. 13.

Author-reported, the prototype head adds 0.6 to 1.2 mIoU on ADE20K over four baselines that the authors reproduced with the MMSegmentation recipe 2203.15102 tab. 1. On ADE20K-847 with SegFormer, it reaches 19.4 mIoU with 27.37 M parameters, against 17.7 for the parametric head with 33.96 M; the table counts learnable parameters only 2203.15102 tab. 4. In the ablation, adding the contrastive term to cross-entropy gives 45.9 against 45.0, and a single prototype per class already gives 45.5 2203.15102 tab. 5. The released code is a different implementation from the paper: it has no temperature, uses a lower compactness weight, and adds projection and normalisation layers before the score, so inference is not pure nearest-prototype retrieval hrnet.py H_48_D_4_proto.json. The only released configuration is HRNet-W48 on Cityscapes, and the paper does not state how the prototypes are initialised 2203.15102 §5.1. Card: Rethinking Semantic Segmentation: A Prototype View. See also prototype learning.

Label-free segmentation: distilling a frozen self-supervised backbone (STEGO)

STEGO uses no labels at all. It keeps a self-supervised DINO ViT-Base frozen, takes its patch features as given, and trains a small head so that the head's per-pixel codes reproduce the backbone's feature correspondences 2203.08414 §3.1 2203.08414 §3.2. The codes are then clustered into classes and refined with a fully connected CRF 2203.08414 §3.3. The loss is applied to three kinds of pair: an image with itself, with one of its seven nearest neighbours, and with a shuffled batch partner 2203.08414 §3.2 2203.08414 §3.3. Clamping the codes at zero before the correlation is part of the method and not a detail; the paper's ViT-S ablation drops unsupervised mIoU on CocoStuff-27 from 24.5 to 10.3 without it 2203.08414 §4.3 2203.08414 app. Additional Ablation Study.

Author-reported on the 27-class CocoStuff validation set, the ViT-B model reaches 28.2 unsupervised mIoU and 41.0 linear-probe mIoU, against 14.4 and 14.8 for PiCIE + H 2203.08414 §4.2. On Cityscapes-27 it reaches 21.0 unsupervised mIoU, against 12.3 for PiCIE 2203.08414 §4.2. The paper reports under two hours of training on one V100 2203.08414 §3.3. These are label-free numbers from a fixed 27-class ontology that the authors call partly arbitrary, so they do not compare directly with supervised mIoU. Card: Unsupervised Semantic Segmentation by Distilling Feature Correspondences.

Domain generalization from one source: selective whitening (RobustNet)

A segmenter trained on synthetic street scenes often degrades on real scenes with other weather, lighting or cities. RobustNet addresses this without target images. It adds a training-only loss to three instance-normalization layers of a DeepLabV3+ 2103.15597 §4.4 2103.15597 app. A.3. Instance normalization standardises each channel but leaves the correlations between channels, which the paper ties to style such as colour and blur 2103.15597 §1 2103.15597 §2. The loss penalises only the covariance entries that move most under colour jitter and Gaussian blur. The entries are found once, after five warm-up epochs, and grouped with k-means at k = 3 2103.15597 §4.3 2103.15597 tab. 7.

Author-reported, trained on GTAV with ResNet-50, the method reaches 36.58, 35.20 and 40.33 mIoU on Cityscapes, BDD-100K and Mapillary, against 28.95, 25.14 and 28.18 for the baseline 2103.15597 tab. 1. The numbers are averages of three runs with no spread reported 2103.15597 §5.2.1. Inference time is the same within noise (10.48 ms baseline, 10.43 ms with the loss, one A100 at 2048 x 1024) 2103.15597 tab. 6. The paper does not measure training time 2103.15597 §5.2.3. The released code is BSD-3-Clause, but its Cityscapes script uses k = 2 and it adds an auxiliary loss the paper's equation omits, so check its card before reproducing the table 2103.15597 app. A.3. Card: RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening. See also domain generalization and feature whitening.

Unsupervised domain adaptation: structure codes and entropy-free losses

When labels exist only in a synthetic source domain (GTA5), two papers adapt the segmenter to Cityscapes without target labels.

• DISE splits every image into a domain-invariant structure code, used for segmentation, and a domain-specific texture code. It is trained with reconstruction, texture-swap translation and adversarial losses; a swapped image keeps the source labels and so gives the target domain extra supervision 1903.12212 §3.1 1903.12212 §3.2. It reaches 45.4 mIoU, against 42.4 for the output-space adaptation of Tsai et al. 1903.12212 tab. 2. Its ablation runs from 39.8 (source only) through 42.6 and 44.1 to 45.4 with label transfer 1903.12212 tab. 4. Training takes 88 GPU hours on one V100 1903.12212 §3.3. Card: All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation.
• Maximum squares loss replaces the entropy penalty on unlabelled target pixels with a negative sum of squared probabilities 1909.13589 §3.2 1909.13589 eq. 6. Two additions are optional: image-wise class weights and a low-level head trained on confident labels 1909.13589 §3.3 1909.13589 §3.4. With DeepLabv2 and ResNet-101 it reaches 46.4 mIoU on GTA5 to Cityscapes, against 42.2 for the authors' entropy-minimisation run 1909.13589 tab. 2. The same loss transfers to Cityscapes to NTHU cross-city adaptation, with 54.5 on Rome against 53.9 without the weights 1909.13589 tab. 6. Card: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss.

Both papers use the same benchmark and backbone family, but their baselines were partly reproduced by the authors, so the headline numbers are not directly comparable (see Contradictions).

Aerial imagery: translate the source images. A different route changes the source images instead of the network. A CycleGAN-style translator, trained without paired images, restyles labelled Potsdam patches to the Vaihingen sensor look. A BiSeNet trained on Potsdam is then fine-tuned on the restyled patches with their original labels 1905.03198 §4.2 1905.03198 §4.3. Overall accuracy rises from 0.35 to 0.52, with the largest gains on Building (0.23 to 0.71) and Tree (0.06 to 0.51) 1905.03198 tab. 3 1905.03198 tab. 4. The paper says the other classes stay the same, but its table shows Low vegetation falling from 0.38 to 0.27 1905.03198 §6 1905.03198 tab. 4. The best epoch is picked on the target test set, and translation quality is checked by eye only 1905.03198 §5.2.4 1905.03198 §5.2.3. Card: Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images. See also domain adaptation and image-to-image translation.

Continual segmentation: new classes, no stored images

PLOP adds classes in steps without keeping old images. It combines a multi-scale feature distillation term (Local POD) with entropy-thresholded pseudo-labels for background pixels 2011.11390 §4.2. On Pascal VOC 2012 in the 15-1 setting (fifteen classes, then one added per step) it reaches 54.64 mIoU over all classes, against 29.70 for MiB (excerpted from its paper) 2011.11390 §4.2. On ADE20K 100-50 the gap is small, 32.94 against 32.79 2011.11390 §4.2. Joint training, which is not continual, reaches 77.40 on VOC 2011.11390 §4.2. Card: PLOP: Learning without Forgetting for Continual Semantic Segmentation.

Long-tail calibration of the classifier (DisAlign)

Distribution Alignment targets skewed label sets, where a few classes dominate the pixels. It freezes the backbone and the original scores, then learns a per-class scale and offset. An input-dependent gate blends the adjusted scores with the originals 2103.16370 §3.2.1 2103.16370 eq. 4 2103.16370 eq. 5. Training uses cross-entropy with class weights proportional to inverse frequency raised to a power rho 2103.16370 §3.2.2 2103.16370 eq. 11. For segmentation the same formula is applied per pixel, with rho set to 0.3 in the paper 2103.16370 §4.2 2103.16370 app. B.2. The authors note that resampling is hard to apply to segmentation and detection, while re-weighting can be used there 2103.16370 §3.3.

Author-reported on ADE20K, mIoU rises from 38.1 to 40.1 for FCN with ResNet-50, and from 46.4 to 47.1 for DeepLabV3+ with ResNet-101 2103.16370 tab. 6. The best segmentation row is DeepLabV3+ with ResNeSt-101, at 47.8 against 47.3 2103.16370 tab. 6. The active tables report no seeds or spread, so gains of a few tenths of a point are not established 2103.16370 §4. The released segmentation config uses rho 0.2 and a 4,000-iteration second stage, while the paper states 0.3 and 8k disalign_fcn_r50-d8_512x512_160k_ade20k.py disalign_fcn_r50-d8_512x512_160k_ade20k.py. Card: Distribution Alignment: A Unified Framework for Long-tail Visual Recognition. See also long-tailed recognition and class-balanced loss.

Weak and open-vocabulary segmentation

• Weak supervision from image labels (ReCAM). ReCAM keeps a converged multi-label classifier and adds a second fully connected layer trained with softmax cross-entropy on single-class features, which are the backbone map weighted by each class's normalised CAM and then pooled 2203.00962 §4.1. On PASCAL VOC 2012 seeds it lifts CAM from 48.8 to 54.8 mIoU, and with IRN refinement from 66.3 to 70.9 2203.00962 tab. 2. Its cost is close to the base method in the ResNet-50 setting (1.0 to 1.9 units) 2203.00962 tab. 2. Card: Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation.

• Open vocabulary (OpenSeg). OpenSeg represents an image as mask proposals with features, not as a dense map alone. It is trained with a Dice loss on class-agnostic COCO masks plus a caption grounding loss 2112.12143 §3.3. With EfficientNet-B7 and prompt engineering it reaches 72.2 mIoU on PASCAL-20 zero-shot, against 52.3 for LSeg with a ViT-L/16 backbone 2112.12143 tab. 2 2112.12143 §4.3. The ADE-150 and Pascal-Context numbers are much lower (for example 28.6 on ADE-150 for the same model) 2112.12143 tab. 2. The paper has no official code, and some baseline rows are copied from other papers 2112.12143 §4. Card: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. See also open-vocabulary segmentation.

• Frozen CLIP with a side network (SAN). SAN keeps CLIP's ViT frozen and attaches a light side ViT of 8 blocks at width 240, which also takes fused CLIP features 2302.12242 §3 2302.12242 §4.2. The side network outputs mask proposals and per-head attention biases. The biases steer CLIP's last blocks, which then classify each proposal 2302.12242 eq. 1 2302.12242 eq. 2 2302.12242 eq. 3. Training is end to end, through CLIP's recognition blocks, on COCO-Stuff 2302.12242 §3 2302.12242 §4.1. Author-reported mIoU on ADE-150 is 27.5 with ViT-B/16 and 32.1 with ViT-L/14, without the ensemble rows 2302.12242 tab. 2. At 640 x 640 with ViT-B/16 the side network adds 8.4 M trainable parameters and runs at 15.2 FPS, against 147.2 M and 0.8 FPS for OvSeg, which the authors re-implemented 2302.12242 tab. 3. The paper's ensemble rows use a model that is not in the public code, so compare the non-ensemble rows 2302.12242 tab. 2. Card: Side Adapter Network for Open-Vocabulary Semantic Segmentation. Its main baseline, OvSeg, has its own card.

• Text-only supervision (GroupViT). GroupViT learns segments from image-caption pairs and no masks. A ViT-S starts from 196 patches, and learned group tokens merge them into 64 and then 8 segments through a hard assignment that keeps the soft gradient, trained with contrastive image-text losses 2202.11094 §3.1 2202.11094 §3.2. A multi-label term adds three nouns from each caption, placed in short templates, as extra positives 2202.11094 §3.2 2202.11094 §4.1. Labels are matched to the output segments by text similarity, with no fine-tuning. Zero-shot mIoU is 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context after training on about 26M public image-text pairs 2202.11094 tab. 5. The supervised DeiT row reports 53.0 and 35.9 2202.11094 tab. 5. The hard assignment carries much of the result: in the one-stage ablation, removing it drops mIoU from 39.3 to 25.1 with the multi-label term kept 2202.11094 tab. 1. Background is the weak point. On PASCAL Context, classes such as ground, road and wall are grouped but rarely named in captions, so their IoU is about 1.5 2202.11094 app. C.3. Card: GroupViT: Semantic Segmentation Emerges from Text Supervision. The official code is non-commercial, so implement from the paper (see its card).

• Two-stage CLIP labelling of mask proposals (zsseg). A MaskFormer trained on seen classes proposes class-agnostic masks, and a frozen CLIP ViT-B/16 scores each masked crop against text prompts 2112.14757 §4.1 2112.14757 §4.2.1. Each mask is binarised at 0.5, the background is filled with a constant colour, and the crop is resized to 224 by 224 2112.14757 §4.2.1. The two classifiers are combined by default, and the per-mask scores are summed into a pixel map 2112.14757 eq. 2 2112.14757 tab. 7. Author-reported zero-shot hIoU is 37.8 on COCO-Stuff, against 20.9 for FCN, and 77.5 on Pascal VOC, against 50.7 for FCN and 39.7 for CaGNet, the best listed prior without self-training 2112.14757 tab. 2 2112.14757 tab. 3. Direct CLIP classification of MaskFormer masks reaches 28.2 hIoU, against 11.0 for Selective Search and 10.9 for GPB-UCM proposals 2112.14757 tab. 5. The paper names no ensemble weight; the released configs use 0.8 for COCO-Stuff and 0.7 for VOC zero_shot_maskformer_R50_bs32_60k.yaml. The "+ST" rows have no procedure in the paper, and the released evaluator is the only description pseudo_sem_seg_evaluation.py. Card: A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model.

Segmentation as a generated output

FashionGAN generates a segmentation map as an intermediate step, then renders clothing texture region by region from a text description 1710.07346 §3.1. The map has seven labels and is generated from an 8x8 coarse map of the wearer that merges all clothing into one class 1710.07346 §3.2. Its quality is judged indirectly: attribute detection on the generated images reaches mAP 82.6, against 71.8 to 74.7 for the baselines and 86.8 on real photos 1710.07346 tab. 1. In a user study, participants labelled the generated map as real in 42% of cases, which the authors read as near chance 1710.07346 §4.3. Card: Be Your Own Prada: Fashion Synthesis with Structural Coherence.

PixelLM takes the idea further: a multimodal language model answers in text and emits one mask per named target. For each target the model writes a set of learned codebook tokens, and a small pixel decoder turns their hidden states into a mask, fusing them across scales and gating each scale by the mask from the coarser one 2312.02228 §3.1 2312.02228 eq. 1. Training uses text cross-entropy, a refinement loss and a DICE term 2312.02228 §3.2. On the MUSE benchmark, the 7B model reaches 39.2 gIoU and 46.3 cIoU, against 12.8 and 27.1 for LISA-7B 2312.02228 tab. 1. LISA is higher on refCOCO val and testB 2312.02228 tab. 3. The model is a LoRA-tuned LLaVA with a frozen CLIP encoder, trained for about 1.5 to 2 days on eight A100 GPUs 2312.02228 §5.1. MUSE was built with GPT-4V, so rebuilding it needs API access 2312.02228 §4.2. Card: PixelLM: Pixel Reasoning with Large Multimodal Model.

Few-shot, in-context and promptable models

Three papers change what the user supplies at test time instead of retraining.

Few-shot cost aggregation (VAT). VAT builds the full correlation volume between query and support positions, then aggregates it with a 4D convolutional Swin Transformer in a three-level pyramid, where each coarser level guides the next finer one. A decoder with appearance cues then produces the mask 2207.10866 §3.3 2207.10866 §3.4 2207.10866 §3.5. Small overlapping 4D convolutions come before the window attention, so window edges do not cut off local context 2207.10866 §3.4.2. Backbones are frozen ImageNet ResNets 2207.10866 §4.1. With ResNet-101, 1-shot mIoU on PASCAL-5i is 67.9 against 66.2 for HSNet, and on FSS-1000 it is 90.3 against 86.5 2207.10866 tab. 1 2207.10866 tab. 3. The volume is large: a standard transformer over it needed about 84 GB per batch, against 3.8 GB for VAT's aggregator 2207.10866 tab. 5. The paper says it uses no augmentation and divides the vote sum by K, but the released configs differ (see Contradictions). Card: Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation. See also cost aggregation and few-shot learning.

In-context medical segmentation (UniverSeg). UniverSeg segments a new medical task without any training. A fully convolutional network takes the query image and a support set of image-label pairs, and the two streams exchange information in each block through a cross-convolution 2304.06131 §3.1. It is trained once on MegaMedical, 53 datasets, with task-level augmentation and synthetic tasks 2304.06131 §4 2304.06131 app. C 2304.06131 app. D. On six held-out datasets its mean Dice is 71.8 at 1.18 M parameters, against 41.8 to 50.1 for the few-shot baselines and 84.4 for per-task nnU-Net 2304.06131 tab. 1. Dice rises with support size and levels off near 64 pairs, and averaging predictions over several support draws helps 2304.06131 §5.3 2304.06131 fig. 7. The cost grows with the number of support pairs. The training code is not released, and the released model differs from the paper in depth and resampling (see its card). Card: UniverSeg: Universal Medical Image Segmentation. See also in-context learning.

Promptable 3D segmentation (SAM-Med3D). SAM-Med3D applies the promptable design of SAM to whole volumes. It has a 3D ViT image encoder, a point-prompt encoder and a 3D mask decoder, trained from scratch in two stages 2310.15161 §4.1 2310.15161 §4.2. One click yields a 3D mask. Author-reported overall Dice is 76.27% with one 3D point, against 36.83% for SAM-Med2D prompted on each slice, and 80.71% with ten points 2310.15161 tab. 3. In the authors' pre-experiment, training from scratch is the most balanced across seen and unseen targets (55.79 and 34.63), against 69.13 and 18.43 for fine-tuning SAM 2310.15161 tab. 2. The release does not include the paper's sliding-window inference for large volumes, and its training differs from the text (see its card) 2310.15161 §4.3. Card: SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images. See also promptable segmentation.

Self-supervised per-pixel features (VADeR)

VADeR is a pre-training method, not a segmenter. It learns one feature per pixel without labels. Two augmented views of an image give known pixel correspondences through their crop and resize geometry, and a contrastive loss pulls matched pixels together against a queue of pixels from other images 2011.05499 §3.1 2011.05499 §3.2. The encoder starts from MoCo v2 and the decoder from random weights 2011.05499 §3.2. With a linear 1x1 head on frozen features, the authors report 56.7 mIoU on VOC, against 54.4 for ImageNet-supervised ResNet-50 and 43.0 for MoCo 2011.05499 tab. 1. On Cityscapes the same frozen features score 44.3, against 47.1 for supervised pre-training, so the VOC lead does not carry over to every dense task 2011.05499 tab. 1. The tables average five trials without standard deviations 2011.05499 §4.1. No official code was found (see its card). Card: Unsupervised Learning of Dense Visual Representations. See also self-supervised learning.

Diffusion models: the mask as a denoising chain

MedSegDiff generates the mask with a diffusion probabilistic model. The reverse chain starts from Gaussian noise in the mask channel and denoises it in steps, conditioned on the scan 2211.00611 §2 2211.00611 eq. 1. Two additions carry the method. Dynamic conditional encoding fuses the current noisy mask into the image encoder at every step 2211.00611 §2.1 2211.00611 eq. 3, and FF-Parser is a learned Fourier-domain filter on the skip features 2211.00611 §2.2 2211.00611 eq. 4. Inference draws 25 samples of 100 steps each and fuses them with STAPLE 2211.00611 §3.2, which is many network evaluations per image.

Author-reported Dice for MedSegDiff++ is 87.5 on optic cup, 90.5 on brain tumour and 86.6 on thyroid nodule. In the same table nnUNet reaches 84.9, 88.2 and 84.2 2211.00611 tab. 1. The ablation credits dynamic conditioning with 2.1, 1.6 and 1.8 Dice points and FF-Parser with a further 0.8, 0.7 and 0.7, on the same three tasks 2211.00611 tab. 2; the model size used for the ablation is not stated. The paper's text gives different gains in places (see Contradictions). The official code differs from the paper in its defaults and in parts of the architecture, so a reimplementation from the paper alone will not match the reported numbers 2211.00611 §3.2. Card: MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model.

SegDiff is a diffusion model for masks on street scenes, aerial images and microscopy. Its noise predictor passes the noisy mask through one convolution and the image through a separate residual dense encoder, adds the two feature maps, and feeds the sum to a U-Net 2112.00390 §4 2112.00390 eq. 17. It is trained from random initialisation, with no pretrained backbone 2112.00390 §5. The mask is the average of several independently sampled reverse chains, then thresholded, and the paper uses 30 chains in its main experiments 2112.00390 §4.1. Cityscapes is scored per object crop, not per image 2112.00390 §5.

Author-reported: mIoU 76.44 on Cityscapes at 15% box expansion, against 73.70 for Spline-GCN 2112.00390 tab. 1; F1 95.14 on Vaihingen buildings, against 94.80 for Deep contour 2112.00390 tab. 2; Dice 81.59 on MoNuSeg, against 79.55 for MedT 2112.00390 tab. 3. The chain count is the main cost, since each chain runs a full set of reverse steps. The released code hard-codes 9 chains and differs from the paper's noise schedule, and it has no LICENSE file, so its code cannot be reused (see its card). Card: SegDiff: Image Segmentation with Diffusion Probabilistic Models.

A diffusion model can also act as a fixed prior at inference time. Graikos et al. train a denoiser independently of the task and add a differentiable constraint while sampling. For land cover the constraint has two terms: mutual information between predicted labels and a local colour clustering, and a KL divergence to weak labels in blocks 2206.09012 §4 2206.09012 app. B.3. Training uses Pittsburgh patches, and the other EnviroAtlas cities are the test sets 2206.09012 tab. 1. On Durham the fine-tuned variant reaches 79.8% accuracy and 46.4 IoU, against 78.9% and 47.9 for a supervised model trained on weak labels. On Phoenix the implicit-posterior baseline is ahead on both metrics (76.2% and 46.0, against 69.6% and 32.4) 2206.09012 tab. 1. The claim is therefore comparable accuracy, not uniformly better results. Card: Diffusion models as plug-and-play priors.

A pretrained diffusion model can also serve as a feature extractor for a pixel classifier. Each labelled image is noised to steps 50, 150 and 250, and the decoder activations of five UNet blocks are upsampled to image size and concatenated into 8448 values per pixel 2112.03126 §3.2. An ensemble of ten small MLPs is trained on the pixels of the few annotated images, and each test pixel takes the majority vote of the ensemble 2112.03126 §3.2 2112.03126 app. C. The authors find later diffusion steps more semantic and middle decoder blocks most informative, and they keep the same steps and blocks for every dataset 2112.03126 §3.1 2112.03126 §3.2.

Author-reported mean IoU, averaged over five data splits, is the highest in every column of the main table: 49.4 on Bedroom-28 against 47.9 for DatasetDDPM and 45.0 for MAE, and 59.1 on FFHQ-34 against 58.8 for MAE 2112.03126 tab. 2. The margin over MAE is small on FFHQ-34 (59.1 against 58.8) and Cat-15 (53.7 against 52.4) 2112.03126 tab. 2. With fewer labels the score drops, to 38.2 on Bedroom-28 with 10 images against 49.4 with 40 2112.03126 tab. 4. The method needs a diffusion model trained on the target domain, which the authors expect to be hard for complex domains such as ImageNet or MSCOCO 2112.03126 §5. Each image yields 256 x 256 x 8448 float32 values, which is about 2.2 GB by calculation from the stated sizes, so feature storage is the practical bottleneck 2112.03126 §3.2. The paper says each MLP trains for about four epochs, which the released code does not do (it stops early), so the training recipe must be fixed by the implementer 2112.03126 app. C. Card: Label-Efficient Semantic Segmentation with Diffusion Models. See also few-shot learning.

Semantic SLAM with Gaussian splats

GS3LAM keeps a dense semantic map inside a set of 3D Gaussians. Each Gaussian carries a position, a shape, a colour, an opacity and a low-dimensional semantic feature, which a learned layer decodes to class scores with a softmax 2603.27781 §3.2 2603.27781 eq. 8. Tracking fits one camera pose per frame with the map frozen. Mapping refines the map on the current frame and on randomly drawn keyframes 2603.27781 §3.4 2603.27781 §3.3.3. A scale regulariser keeps Gaussian sizes in range. The paper names it as a contribution, but the released Replica config switches it off (see its card) 2603.27781 §3.3.2. Author-reported on Replica: rendering PSNR 36.26 over eight scenes 2603.27781 tab. 2, mIoU 96.63 over four scenes 2603.27781 tab. 4, and tracking ATE 0.37 cm, against 0.36 cm for SplaTAM 2603.27781 tab. 3. Card: GS3LAM: Gaussian Semantic Splatting SLAM. See also SLAM and Gaussian splatting.

Distilled feature fields in 3D Gaussians (Feature 3DGS)

Feature 3DGS gives each 3D Gaussian a feature vector and distils it from a 2D teacher, LSeg or SAM, so that text and prompts can select regions in an offline 3D scene 2312.03203 §3.2. Colour and feature are rendered together, and the loss adds an L1 feature term to the RGB loss with weight 1.0 2312.03203 eq. 4. An optional speed-up renders fewer feature channels and lifts them back with a learned 1x1 convolution 2312.03203 §3.2. A text query is matched to features by cosine similarity and a softmax over labels 2312.03203 eq. 5 2312.03203 eq. 6. On Replica, author-reported mIoU is 0.782 at 14.55 FPS with the speed-up and 0.787 at 6.84 FPS without it, against 0.636 at 5.38 FPS for NeRF-DFF 2312.03203 tab. 2. The paper says the speed-up does not affect performance 2312.03203 §3.2, but its own dimension sweep drops LSeg features to mIoU 0.354 at dimension 8 2312.03203 tab. A. The quantitative results are on Replica only. The code licence is non-commercial research use only, and the card records it as such. Card: Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields. See also Gaussian splatting.

LiDAR semantic SLAM: labels decide which surfels to trust (SuMa++)

SuMa++ adds per-point semantics to a surfel-based LiDAR mapping pipeline. A segmenter labels each point of the scan's spherical projection, and the labels are used at three points: a flood-fill cleans the label image, the map update penalises surfels whose stored class conflicts with a movable class, and the ICP residuals are weighted by semantic compatibility 2105.11320 §III-C 2105.11320 §III-D 2105.11320 §III-E 2105.11320 §III-F. Deleting every movable surfel instead is not a fix: parked cars help registration, and on urban sequences the deletion variant diverges 2105.11320 §IV-B 2105.11320 tab. II.

Author-reported, on KITTI road sequences 30 to 41 the average relative translation error falls from 6.13% (SuMa) to 1.46% (SuMa++), and on odometry training sequences 00 to 10 it falls from 0.83% to 0.70% 2105.11320 tab. I 2105.11320 tab. II. On sequence 35, a highway, the error falls from 26.8% to 1.11% 2105.11320 tab. I. The training-sequence numbers are in-sample, because the segmenter was trained on sequences 00 to 10 except 08 2105.11320 §IV. The method needs a labelled training set and a trained segmenter, and the evaluation is on KITTI only 2105.11320 §IV 2105.11320 §V. Card: SuMa++: Efficient LiDAR-based Semantic SLAM. See also SLAM and iterative closest point.

Distillation for dense prediction

DIST replaces the exact KL match in knowledge distillation with a Pearson-correlation loss, applied across classes within each instance and across instances within each class 2205.10536 §3.1 2205.10536 §3.2. On Cityscapes with a DeepLabV3-R18 student and an R101 teacher it reaches 77.10 mIoU, against 74.21 for the student alone and 75.42 for SKD 2205.10536 tab. 7. On ImageNet the gain is largest with the strongest teacher tested (ResNet-18 student: +0.86 with ResNet-34, +1.12 with ResNet-152) 2205.10536 tab. 3. Card: Knowledge Distillation from A Stronger Teacher. See also knowledge distillation.

Building footprints from satellite imagery

Continent-scale building mapping uses a U-Net with a ResNet-50-v2 encoder and residual decoder blocks. Training combines an edge-weighted cross-entropy with a focal Tversky term, and inference thresholds the averaged confidence and takes connected components as instances 2107.12283 §4 2107.12283 §5 2107.12283 §10. Mixup on the cross-entropy term, ImageNet initialisation and soft-label self-training are the other main additions 2107.12283 §6 2107.12283 §7 2107.12283 §8. Author-reported, mixup and edge weighting have the largest ablation effects, and self-training adds +0.057 mAP on average 2107.12283 §11 2107.12283 fig. 8. The pipeline produced 516M polygons after deduplication 2107.12283 §12. Card: Continental-Scale Building Detection from High Resolution Satellite Imagery. See also instance segmentation.

Infrared small targets: nested skips and one centroid per target (DNA-Net)

Single-frame infrared targets are often one to a few tens of pixels wide, and pooling can remove a one-pixel target before the output stage 2106.00487 §1 2106.00487 §3.2.1. DNA-Net nests U-shaped sub-networks densely. Each node takes earlier nodes at its own level, the level below in the previous column, and the level above in its own column 2106.00487 eq. 2 2106.00487 §3.2. Each node block ends with channel and then spatial attention, and the output fuses the top nodes of all five levels 2106.00487 §3.2.3 2106.00487 eq. 7 2106.00487 §3.3. Connected components of the thresholded mask are reduced to centroids 2106.00487 §3.4 2106.00487 eq. 8.

On NUDT-SIRST, a synthetic set of 1327 images, the ResNet-18 variant reports IoU 87.09, Pd 98.73 and Fa 4.223 (IoU and Pd in units of 10^-2, Fa in units of 10^-6), against IoU 81.40, Pd 96.51 and Fa 9.261 for ALCNet 2106.00487 tab. II. On NUAA-SIRST, a real set, it reports the best Pd and Fa, while its IoU of 77.47 is second to the ResNet-34 variant at 77.54 2106.00487 tab. II. Pd counts matched targets under a 3-pixel centroid tolerance, Fa counts false-alarm pixels, and the reported Pd moves with that tolerance: 98.51 at 2 pixels and 98.73 at 3 2106.00487 §5.1 2106.00487 tab. IV. Every CNN baseline was retrained by the authors on the same split, so the numbers are not the baselines' published results 2106.00487 §5.3. Training uses Adagrad with a learning rate of 0.05, batch 16 and 1500 epochs 2106.00487 §5.2. The paper does not give the probabilities that choose target types, or the full blur sigma list, for the synthetic data 2106.00487 §4.2, so that data cannot be regenerated from the paper alone. Card: Dense Nested Attention Network for Infrared Small Target Detection.

Datasets, benchmarks and surveys

Synscapes is a synthetic street-scene set of 25,000 path-traced images. Each image has a Cityscapes-style class map, an instance map, a depth map and per-actor metadata 1810.08705 §3 1810.08705 §3.2. Scenario parameters are sampled independently for each image, which allows error to be measured against distance, occlusion and orientation 1810.08705 §6. Author-reported, a Cityscapes-trained DeepLab v3+ scores 63.64 mean IoU on the Synscapes validation split, against 45.84 on the GTA-based set 1810.08705 tab. 1. Pre-training on Synscapes and then fine-tuning on Cityscapes gives 78.85, against 76.56 for Cityscapes alone 1810.08705 tab. 4. The authors attribute the gap to realism, but the GTA-based set is the most varied in models and building types, so realism is not isolated 1810.08705 §7. The paper's CC BY licence covers the paper, not the dataset, and the paper does not state the dataset's licence or download location 1810.08705 §3. Card: Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing. See also sim-to-real transfer.

Replica is a set of 18 indoor 3D scenes with dense meshes, 16-bit HDR textures, semantic labels in 88 classes and mirror and glass reflectors 1906.05797 §4 1906.05797 tab. 1. It is a data release, not a method, and the paper reports no learning benchmark, so it does not show that models trained on it transfer to real images 1906.05797 abstract. The paper is CC BY 4.0, but the scene data is covered by the repository's research terms, which allow only non-commercial research or education. Check those terms before any commercial use. Card: The Replica Dataset: A Digital Replica of Indoor Spaces. See also robot simulation.

RELLIS-3D is an off-road dataset with image and LiDAR labels in 20 classes. The authors report 51.55 mIoU for HRNet+OCR and 52.92 for Gated-SCNN on images, and 43.07 for SalsaNext and 19.97 for KPConv on point clouds 2011.12954 tab. 2. Licensing: the card records that the repository states non-commercial terms for the data and the code. Do not use this dataset or its code in a commercial context without the authors' permission. Card: RELLIS-3D Dataset.

A benchmark for 3D medical segmentation, nnU-Net Revisited, trains CNN, transformer and Mamba models on six public datasets (BTCV, ACDC, LiTS, BraTS2021, KiTS2023 and AMOS2022). Each model is trained from scratch on one 40 GB A100, with five-fold cross-validation on nnU-Net's splits 2404.09556 §3.1 2404.09556 §3.2. The authors argue that many architecture gains come from confounds such as extra data, pre-training, a weaker baseline or a small split 2404.09556 §2.1 2404.09556 §2.2. Under this protocol, the CNN U-Nets built in the nnU-Net framework lead. MedNeXt L with kernel size 5 has the best mean Dice on BTCV, LiTS, BraTS and AMOS, MedNeXt L with kernel size 3 on ACDC, and nnU-Net ResEnc XL on KiTS2023 2404.09556 tab. 1. The transformer models (SwinUNETR, nnFormer and CoTr) do not match the original nnU-Net, and the U-Mamba gain over the "No-Mamba Base" ablation disappears; the card attributes the earlier gain to the residual encoder 2404.09556 §4 2404.09556 tab. 1. The dataset suitability ratio, inter-method over intra-method spread, is highest for AMOS, KiTS and ACDC when all methods are included, and it falls for BTCV and ACDC without the Auto3DSeg runs 2404.09556 tab. 3. The ranking therefore depends on the method set. Only the nnU-Net code was checked; the other methods' code was not 2404.09556 §3.1. Card: nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation.

A survey of lane topology for HD maps, Yao et al., uses segmentation as one step in its aerial route. A segmentation network labels road pixels, and morphological post-processing then turns the mask into a road graph 2504.01989 §2.2. The paper does not run any method itself, and its metrics are road-graph scores (APLS, TLTS) rather than mIoU 2504.01989 §3.2. It reports the aerial route's weakness as occlusion by buildings and trees, and it calls aerial methods unsuitable for automatic HD-map construction 2504.01989 §2.3.

The anomaly-detection survey is not a segmentation benchmark and reports no experiments of its own. Its camera section lists confidence scores from segmentation networks, such as Bayesian SegNet and standardized max logits, among the baselines for unusual road scenes 2204.07974 §2. Pixel-level anomalies are outside its scope 2204.07974 §1. Card: Anomaly Detection in Autonomous Driving: A Survey.

A review, Deep Semantic Segmentation of Natural and Medical Images, groups deep segmentation work into architectures, losses, synthesis, weak supervision and multi-task models. It runs no experiments of its own 1910.07655 §1 1910.07655 §10. Its loss section is the most practical part. In a worked example with one false negative, Dice gives 0.96 on a large object and 0.66 on a small one, while cross-entropy gives 0.83 for both. The review advises pairing overlap losses with cross-entropy 1910.07655 §4.4 1910.07655 §10.3. These values are illustrative, not benchmark results. Its VOC 2012 mean-IoU table collects numbers from other papers, which use different backbones and protocols, so it is not a ranking 1910.07655 tab. 2.

Three more surveys cover segmentation within a wider scope. A review of deep learning on UAV imagery codes 232 articles and proceedings from 2016 to 2021 by network, task, sensor and context. Its tallies put object detection at 53.9% and segmentation at 40.7% of the studies 2101.10861 §3. It runs no experiments and its percentages are the authors' own 2101.10861 §3. Card: A Review on Deep Learning in UAV Remote Sensing.

A survey of deep learning for weed detection covers 70 papers from 2016 to August 2020 and lists 19 public datasets, most of them specific to one site or crop 2103.01415 §4 2103.01415 §6.3. Its accuracy figures come from different crops, weed species and metrics, and the survey does not rank them 2103.01415 §11. Card: A Survey of Deep Learning Techniques for Weed Detection from Images.

A review of vision transformers in medical imaging groups segmentation models by where the transformer sits in the network 2301.03505 §2.2. Its Synapse table lists MERIT at 84.90 Dice, but the review does not say whether each row was copied or re-run, so the rows may not share a protocol 2301.03505 tab. 6. Card: Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review.

Augmentation survey with segmentation runs

The survey sorts image augmentation into manipulation, erasing, mixing, automated policy search, feature-space and GAN-based methods, and notes that the label must change with the image 2204.08610 §1 2204.08610 §2.1. Experiments cover only the first three groups 2204.08610 §4.1. On PASCAL VOC, author-reported mean IoU moves from 75.32 to 75.75 for DeepLabV3+ and from 71.65 to 74.26 for ISANet 2204.08610 tab. 2. Each result is a single run with no variance 2204.08610 §4. The printed change column disagrees with the recomputed differences in several rows, so recompute from the two value columns 2204.08610 tab. 2 2204.08610 tab. 4. No code is released, and the optimiser, epochs, batch size and segmentation backbone are not stated 2204.08610 tab. 1. Card: Image Data Augmentation for Deep Learning: A Survey.

Mid-level labels rendered from 3D scans (Omnidata)

Omnidata is a pipeline that renders mid-level vision labels from textured 3D scans. It places cameras by Poisson-disc sampling, samples points, and keeps only points seen by at least three cameras 2110.04994 §3.1. It renders 21 cues, including depth and surface normals, each taking 1 to 4 seconds per 512 x 512 image on a CPU 2110.04994 §3.1. The starter set has 14,601,449 images from 2,414 scenes 2110.04994 §4.2 2110.04994 tab. 1. Zero-shot depth on NYU reaches an L1 error of 0.2878, against 0.3838 for MiDaS v3 2110.04994 tab. 2. The full cue pipeline is only in the supplement, and the 21 cues are shown in a figure only 2110.04994 §3.1. The starter dataset and weights are for non-commercial research and education, and the card records that licence. Card: Omnidata.

Scope note: a driving agent that takes segmentation images (InDRiVE)

InDRiVE is listed here because its input is a semantic-segmentation image, not because it segments. It pretrains a DreamerV3-style world model in CARLA with ensemble disagreement as an intrinsic reward and no task reward, then tests it zero-shot on an unseen town 2512.18850 §3.2.1 2512.18850 §4.1. The card gives zero-shot success rates of 75.65 on Town01 and 73.20 on Town02 2512.18850 tab. 6. The repository licence is non-commercial, apart from the dreamerv3 folder, so the code is not commercial-safe. Card: InDRiVE.

Anomaly detection by resynthesis

Unknown objects on a road are often given a known label by a segmenter. The resynthesis approach repaints the image from the predicted label map with a pix2pixHD generator, then scores the pixels where the repainting disagrees with the photo 1904.07595 §3 1904.07595 fig. 2. A discrepancy network reads the photo, the repainting and the labels. It is trained only on known classes: a random subset of object instances is given a different known label, the generator repaints from that altered map, and the changed pixels become the positive target 1904.07595 §3.2. The released config uses half the instances (see the card). Author-reported AUROC is 0.79 to 0.93 on Lost and Found and a 60-image road set, above a Bayesian-uncertainty baseline and an RBM reconstruction baseline in every setting 1904.07595 fig. 5. The same comparison, with a HOG-distance and logistic-regression detector, also flags adversarial inputs 1904.07595 §3.3. The method needs a generator trained on the target domain, and the authors name false positives as the main open issue 1904.07595 §5. Its released code is GPL-3.0-or-later. Card: Detecting the Unexpected via Image Resynthesis. See also anomaly detection.

When to use which

All numbers are author-reported. mIoU (%) on the val split unless stated otherwise.

Regularisation for the ViT models: encoder dropout off, stochastic depth around 0.1. On Seg-S/16, dropout 0.2 cuts ADE20K mIoU from 45.37 to 36.63 2105.05633 tab. 2.

Implementation sketch

The ViT pipeline:

Input: image (H0, W0, 3), K classes, patch size P, pre-trained ViT with L layers

1. Pad the image so H and W are multiples of P (Segmenter code pads bottom/right
   and crops the prediction back).
2. Patch-embed: P x P patches -> linear projection -> N = HW/P^2 tokens of width D.
3. Add learned position embeddings, bilinearly resized from the pre-training grid
   to (H/P, W/P).
4. Run L pre-norm transformer blocks; keep the outputs of every layer if the
   decoder needs intermediate features (SETR-MLA, auxiliary heads).
5. Decode the (H/P, W/P, D) map to (H/P', W/P', K) scores:
     SETR-Naive : 1x1 conv, BN, ReLU, 1x1 conv to K
     SETR-PUP   : 4 x [conv, 2x upsample]
     SETR-MLA   : fuse M evenly spaced layers top-down, upsample, concatenate
     Seg-Linear : linear D -> K per token
     Seg-Mask   : append K class embeddings, run M = 2 blocks,
                  masks = normalise(patches) . normalise(classes)^T
6. Bilinearly upsample to (H, W); crop padding.
7. Train: per-pixel cross-entropy (+ SETR auxiliary heads on intermediate layers).
   Infer: sliding window at the training crop size, optional multi-scale + flip,
   argmax over K.

Where the variants differ in detail:

• SETR's PUP channel widths, MLA's M and stream order, and the auxiliary loss weight are only shown in a figure or not stated at all 2012.15840 §3.3 2012.15840 §4.1.
• Segmenter's paper applies softmax then layer norm to the upsampled masks; its released code instead applies LayerNorm to the patch-level masks before upsampling and leaves softmax to the loss, and adds learned projections before the scalar product. The card recommends following the code 2105.05633 §3.2.
• GroupViT: the paper's assignment has no scale factor, and the released code adds one, plus an epsilon in the normalisation of the segment update. Its background thresholds (0.9 and 0.5 in the paper) also differ from the config values, so the numbers depend on which rules you follow 2202.11094 §3.1 2202.11094 §4.1. The card lists the code-side details.
• ACNN: the shape term's weight (0.01 in the paper) is described as "chosen experimentally" for the authors' intensity normalisation, so the value may not carry over to other scalings 1705.08302 §2.2.
• Batch size on ADE20K: SETR uses 16, Segmenter 8; both use LR 1e-3 there and 1e-2 on Cityscapes 2012.15840 §4.1 2105.05633 §4.2.
• Cost: global attention over N tokens in every layer. A 768x768 Cityscapes crop at P = 16 is 2,304 tokens per layer; SETR runs at stride 16 rather than the stride 8 of its dilated FCN baseline for GPU memory reasons 2012.15840 §4.1, and Segmenter drops its Cityscapes Large mask decoder to one layer for memory 2105.05633 §4.4.
• STEGO: the released config's defaults are a ViT-Small run, not the paper's ViT-B setting, and the paper's batch size is 32 where the config uses 16. Start from the paper's CocoStuff or Cityscapes row of loss weights and shifts 2203.08414 app. Implementation Details. The paper states the loss as sums, while the code averages each term, which changes the loss scale and so its interaction with the learning rate and loss weights 2203.08414 §3.2 2203.08414 eq. 4.

The U2PL training step, in the paper's terms. The per-class loop is the part most often mis-implemented:

# One U2PL training iteration  [alg. 1] [eq. 1]
# Input: labeled batch (x_l, y_l), unlabeled batch x_u, epoch t of T, K classes
# The teacher is an EMA of the student and receives no gradient

# Teacher class probabilities on unlabeled pixels  [§3.2]
p = softmax(teacher(x_u))
# Per-pixel entropy of those probabilities  [eq. 5]
H = entropy(p)
# Unreliable share shrinks linearly from 20% towards 0  [eq. 7]
alpha = 0.2 * (1 - t / T)
# Threshold so that the top alpha share is unreliable  [§3.2]
gamma = quantile(H, 1 - alpha)
# Argmax pseudo-labels for reliable pixels, ignore the rest  [eq. 6]
pseudo = where(H < gamma, argmax(p), IGNORE)
# Supervised cross-entropy on labeled pixels  [eq. 2]
L_s = cross_entropy(student(x_l), y_l)
# Rescale the unsupervised loss by the share of kept pixels  [eq. 8]
L_u = cross_entropy(student(x_u), pseudo) * (pixels / kept)  # [eq. 3]

for c in classes:
    # Anchors: labeled pixels of c, plus reliable pixels pseudo-labeled c,
    # both with teacher p_c above 0.3  [eq. 9-11]
    anchors = anchor_set(c, y_l, pseudo, p, delta=0.3)
    # Positive: mean teacher feature over the anchors  [eq. 12]
    positive = mean(teacher_feat[anchors])
    # Negatives for c: labeled pixels whose top-3 contains c and not ground
    # truth c; unlabeled unreliable pixels whose rank of c is in [3, 20)
    # [eq. 14-15]
    queue[c].push(negatives_for(c, y_l, p, H, gamma))
    # Sample M anchors, N negatives from the queue, InfoNCE with tau = 0.5  [eq. 4]
    L_c += infonce(sample(student_feat[anchors], M), positive,
                   sample(queue[c], N), tau=0.5)

# Total loss; lambda_c = 0.1 in the paper's table  [eq. 1] [app. tab. A1]
loss = L_s + L_u + lambda_c * L_c

Contradictions

• Cross-paper numbers agree. Segmenter's tables quote SETR as 48.64 / 50.28 on ADE20K, 55.8 on Pascal Context and 82.2 on Cityscapes 2105.05633 tab. 6 2105.05633 tab. 7 2105.05633 tab. 8, matching SETR's own 48.64 / 50.28, 55.83 and 82.15 2012.15840 tab. 4 2012.15840 tab. 5 2012.15840 tab. 6.

• Is the decoder or the encoder the key? SETR's results suggest the decoder choice is dataset-specific (PUP for Cityscapes, MLA for ADE20K) 2012.15840 §4.2; Segmenter's ablations show a linear head already competitive and the encoder size and patch size as the larger levers 2105.05633 tab. 3 2105.05633 tab. 4. The two studies use different decoders, batch sizes and pre-trained weights, so they cannot be compared directly. U2PL uses a CNN backbone, other splits and label fractions, so its numbers do not compare with the ViT tables either.

• Two UDA papers, one benchmark. DISE reports 45.4 on GTA5 to Cityscapes and maximum squares reports 46.4 with its full method 1903.12212 tab. 2 1909.13589 tab. 2. Both compare with an output-space adaptation at 42.4 1903.12212 tab. 2 1909.13589 tab. 2, but the maximum-squares paper reproduces its entropy baseline itself (42.2), and DISE's paper does not state its loss weights while its released code differs from the text 1903.12212 §3.2. The two headline gaps are therefore not verified against one protocol.

• Inconsistencies inside each paper. SETR reports SETR-PUP-DeiT on ADE20K as 46.24 in one table and 46.34 in another, and labels DeiT rows T-Large in tab. 3 but T-Base elsewhere 2012.15840 tab. 3 2012.15840 tab. 4. Segmenter reports Seg-L-Mask/16 as 51.30 SS in tab. 4 and 51.82 SS in tab. 6, without explaining the gap 2105.05633 tab. 4 2105.05633 tab. 6. DISE gives Tsai et al. 42.4 in one table and 42.6 for the same method in another 1903.12212 tab. 2 1903.12212 tab. 4. The maximum-squares text claims a "significant" gain from the multi-level head, while its table shows +0.9 and +1.2 1909.13589 tab. 4; quote the table, not the prose. RELLIS-3D gives KPConv 19.07 mIoU in the text and 19.97 in the table 2011.12954 tab. 2.

• Mixed metrics for nnFormer. Against nnUNet on Synapse, nnFormer has lower HD95 (10.63 vs 10.78) but lower DSC (86.57 vs 86.99) 2109.03201 tab. 5. The paper's text calls the DSC differences on tumour and Synapse not significant 2109.03201 §4.3. Pick the metric that matches the task.

• Benchmark protocol for ColonSegNet. The paper describes its 120 evaluation images as "validation" in one place and as "test" in the figure captions; its code uses the same images for checkpoint selection and for testing 2011.07631 §V-B, so the reported scores are not a clean held-out result.

• Steel micrographs: headline numbers. The 93.94% headline has no stated denominator. §5.1 gives 2262 test objects, but the confusion matrix covers 2205 classified objects plus 48 unsegmented ones 1706.06480 §5.1 1706.06480 tab. 4. The text says augmentation adds 2 percentage points, while the table gives 2.97 1706.06480 §6 1706.06480 tab. 2. The 48.89% baseline is "shown" on the same data, without saying whether it was re-run or copied 1706.06480 §5.3.

• Speed-up figure. Cross-view Transformer states a 4.5x inference speed-up over FIERY 2205.02833 tab. 1; the table's 35 and 8 FPS give about 4.4x. Quote the FPS values.

• MedSegDiff gains, text versus table. The text gives a 2.3% Dice and 2.4% IoU gain over EnsemDiff on brain tumour. Table 1 gives 88.7 to 90.5 Dice and 80.9 to 82.8 IoU, which is 1.8 and 1.9 points 2211.00611 §3.3 2211.00611 tab. 1. The text calls the FF-Parser gain "near 1%", while Table 2 gives 0.7 to 0.8 points 2211.00611 §3.4 2211.00611 tab. 2. Quote the tables.

• Few-shot augmentation and voting. VAT's text says it uses no augmentation and divides the vote sum by K 2207.10866 §4.1 2207.10866 §3.6. Its released configs apply photometric augmentation, and its code normalises votes by the largest count in the image (see its card). Reported numbers depend on which rule was used, so state the choice when comparing.

• Text versus table, within one paper. GS3LAM calls its tracking competitive, but SplaTAM's Replica ATE of 0.36 cm is below its 0.37 cm 2603.27781 tab. 3. Its text gives a 0.039 LPIPS gap to the runner-up, where the table gives 0.030 against GS-SLAM 2603.27781 §4.2 2603.27781 tab. 2. UniverSeg's text gives Dice gains of 7.3 to 34.9 over the best baseline, while its per-dataset table gives 6.4 to 33.0 2304.06131 §5.2 2304.06131 app. E. Quote the tables.

• Abstract versus body. The building pipeline's abstract gives mixup +0.12 and self-training +0.06 mAP. The body gives +0.057 for self-training, and the mixup effect appears only in a figure 2107.12283 fig. 8. Use the body numbers.

• Aerial adaptation, claim versus table. The aerial GAN paper says classes other than Building and Tree are unaffected, but its table shows Low vegetation falling from 0.38 to 0.27 1905.03198 §6 1905.03198 tab. 4.

• TransUNet baseline row. The per-organ DSC values in the R50-AttnUNet row do not average to the printed 75.57; the aorta value of 55.92 looks like the outlier 2102.04306 tab. 1. Check that row before using it as a baseline.

• DNA-Net internal numbers. Table IV gives Fa 4.228 for DNA-Net-ResNet18 and 9.581 for ALCNet under 4 pixels, while Table II gives 4.223 and 9.261 2106.00487 tab. II 2106.00487 tab. IV. The real-data mixing table lists Fa 4.223 for the NUAA row, where Table II gives 2.353 2106.00487 tab. VIII 2106.00487 tab. II. The mixing table also counts 791 synthetic training images, while the released NUDT split has 663 training ids 2106.00487 tab. VIII. Quote Table II and state the centroid tolerance.

• GroupViT multi-label gains, text versus table. The text attributes a 13.1-point gain to hard assignment and a 2.6-point gain to soft assignment 2202.11094 §4.2. Table 1 shows the 13.1-point gain without hard assignment (12.0 to 25.1) and the 2.6-point gain with it (36.7 to 39.3) 2202.11094 tab. 1. Quote the table. The text also puts the 2-stage gain on CC12M+YFCC at about 7% 2202.11094 §4.2, against +15.1 mask mIoU in the table 2202.11094 tab. 3.

• ACNN speed and significance, text versus table. The text says the super-resolution model is faster "by a factor of 5", but the table gives 0.29 s against 0.06 s, about 4.8 times 1705.08302 §3.5 1705.08302 tab. 3. The text calls the ultrasound Dice gain not significant, while the table gives p about 0.05 1705.08302 §3.4 1705.08302 tab. 2.

• RobustNet claim versus its table. The text says ISW beats the other methods on BDD-100K, Mapillary and SYNTHIA when trained on Cityscapes 2103.15597 §5.2.1. In that table, ISW scores 58.64 on Mapillary, below IRW at 59.20 and IW at 58.90 2103.15597 tab. 2. Quote the table.

• HiFormer repeated number. The ResNet-18 row of its backbone ablation and the no-fusion row of its fusion ablation both report Dice 77.15 and HD 16.88, and the paper does not explain the repeat 2207.08518 tab. 6 2207.08518 tab. 7. Confirm the setting before quoting either.

• PVT v2 segmentation sentence. The text compares "PVT-Large" with ResNeXt101-64x4d and reports 48.7 mIoU 2106.13797 §4.3. The table gives 48.7 for PVT v2-B5 and 42.1 for PVT v1-Large 2106.13797 tab. 5. Quote the table.

• Transformers against nnU-Net on brain tumours. The nnFormer card reports DSC 82.02 against 81.87 for nnUNet, and a lower HD95, on its brain-tumour task 2109.03201 tab. 5. The nnU-Net Revisited benchmark ranks nnFormer below the original nnU-Net on BTCV, LiTS, BraTS2021, KiTS2023 and AMOS2022, and above it only on ACDC (92.40 against 91.54) 2404.09556 tab. 1. The cards do not show that the two brain-tumour tasks are the same, and the protocols differ, so this stays open. Quote both with their setup.

• OpenSeg and SAN on ADE-150. OpenSeg reports 28.6 on ADE-150 for EfficientNet-B7 with prompt engineering 2112.12143 tab. 2. SAN reports 27.5 with ViT-B/16 and 32.1 with ViT-L/14 2302.12242 tab. 2. The backbones and training data differ. SAN's table has an OpenSeg row, which its card does not copy, so the two numbers are not from one comparison and do not give a ranking.

• FastFCN's own mIoU values. The text gives 52.10 and 54.03 mIoU on the 59-class PASCAL-Context protocol, while the abstract gives 53.13 for ResNet-101 1903.11816 §4.3. The card does not resolve which is the reported result, so quote the table with its setup.

• PixelLM and LISA on different benchmarks. PixelLM reports 39.2 gIoU against 12.8 for LISA-7B on MUSE 2312.02228 tab. 1, while LISA is higher on refCOCO val and testB [arxiv:2312.02228 tab. 3]. The two benchmarks measure different things, so the results do not give a ranking.

Papers

• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers: SETR, a plain-ViT encoder at fixed 1/16 resolution with Naive, PUP and MLA convolutional decoders.
• Segmenter: Transformer for Semantic Segmentation: a fully transformer pipeline with a linear or class-token mask decoder, plus ablations of model size, patch size, regularisation, data size and pre-training.
• Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels: U2PL, a teacher-student method that keeps unreliable pixels as negatives in per-class queues for a pixel-level contrastive loss, on a ResNet-101 DeepLabv3+.
• FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics: a U-Net-style network with residual blocks and summation skips, for electron-microscopy neuron boundaries.
• Advanced Steel Microstructural Classification by Deep Learning Methods: a pixel-wise FCN with per-object majority voting for steel micrographs, on 21 image pairs.
• Be Your Own Prada: Fashion Synthesis with Structural Coherence: text-guided outfit generation that first generates a clothing segmentation map and then renders each region.
• Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders: a bidirectional convolutional recurrent encoder over Sentinel-2 time series for per-pixel crop classes.
• All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation: DISE, which segments from a domain-invariant structure code and adapts by swapping texture codes.
• Domain Adaptation for Semantic Segmentation with Maximum Squares Loss: a squared-probability target loss with image-wise class weights and a low-level guidance head.
• Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning: a benchmark on Kvasir-SEG plus ColonSegNet, a small fast encoder-decoder.
• PLOP: Learning without Forgetting for Continual Semantic Segmentation: continual segmentation with local pooled-output distillation and entropy-thresholded pseudo-labels.
• RELLIS-3D Dataset: Data, Benchmarks and Analysis: an off-road dataset with image and LiDAR labels and baseline results; non-commercial terms, see Datasets and benchmarks.
• HDMapNet: An Online HD Map Construction and Evaluation Framework: vector map elements from cameras, LiDAR or both, in bird's-eye view.
• nnFormer: Interleaved Transformer for Volumetric Segmentation: a 3D U-shaped transformer with local and global attention and skip attention, for medical volumes.
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels: OpenSeg, mask proposals with caption grounding for zero-shot segmentation.
• Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation: ReCAM, a second classifier layer that sharpens CAM seeds from image-level labels.
• Cross-view Transformers for real-time Map-view Semantic Segmentation: a six-camera transformer that lifts image features into a top-down map with cross-view attention.
• Knowledge Distillation from A Stronger Teacher: DIST, a correlation-based distillation loss that also reports Cityscapes segmentation results.
• Transformers in Remote Sensing: A Survey: a survey that collects reported transformer numbers for remote-sensing tasks, without re-running them.
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences: STEGO, a label-free head on a frozen DINO ViT-B that clusters distilled feature correspondences.
• MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model: a conditional diffusion model for medical masks, with dynamic conditional encoding and FF-Parser.
• Diffusion models as plug-and-play priors: a fixed diffusion prior with a differentiable task constraint at sampling time, with a land-cover segmentation experiment.
• Anomaly Detection in Autonomous Driving: A Survey: a survey of corner-case detection for driving across sensors, with no experiments of its own.
• Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing: 25,000 path-traced synthetic street scenes with per-pixel, instance and depth labels.
• Detecting the Unexpected via Image Resynthesis: flags unknown road objects by repainting the scene from its label map and comparing.
• Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images: GAN restyling of labelled aerial patches, then fine-tuning.
• Unsupervised Learning of Dense Visual Representations: VADeR, per-pixel contrastive features learned from view correspondences.
• HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS: a HarDNet68 encoder with a light RFB and dense-aggregation decoder for polyps.
• Continental-Scale Building Detection from High Resolution Satellite Imagery: U-Net building masks with edge weights, mixup and self-training.
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation: VAT, transformer aggregation of the query-support correlation volume.
• UniverSeg: Universal Medical Image Segmentation: in-context medical segmentation from a support set, with no training for a new task.
• SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images: a promptable 3D ViT for medical volumes, trained from scratch.
• GS3LAM: Gaussian Semantic Splatting SLAM: a semantic Gaussian field with pose tracking and keyframe mapping for RGB-D SLAM.
• Attention U-Net: Learning Where to Look for the Pancreas: additive attention gates on the skip connections of a 3D U-Net, replacing a localisation cascade.
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation: a ResNet-50 stem with a ViT-B/16 encoder and a cascaded upsampler, for CT and MRI.
• Road Extraction by Deep Residual U-Net: a residual U-Net with no cropping for road masks in aerial images, evaluated with a relaxed break-even point.
• Dense Nested Attention Network for Infrared Small Target Detection: nested U-shaped sub-networks with channel and spatial attention, and centroid extraction for infrared targets.
• The Replica Dataset: A Digital Replica of Indoor Spaces: 18 indoor 3D scenes with HDR textures and semantic labels; research-only scene terms.
• ACNN: an autoencoder-code shape penalty for cardiac segmentation and super-resolution, used only during training.
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation: a U-Net with an ImageNet-pretrained VGG11 encoder for binary building masks.
• Label-Efficient Semantic Segmentation with Diffusion Models: per-pixel features from a pretrained DDPM with an ensemble of small MLPs, for few-label segmentation.
• GroupViT: Semantic Segmentation Emerges from Text Supervision: grouping transformer trained on image-caption pairs, with zero-shot label matching.
• A Concise Survey on Lane Topology Reasoning for HD Mapping: a survey of lane topology methods that includes an aerial route built on road-pixel segmentation; no code, no new method.
• Coordinate Attention for Efficient Mobile Network Design: a channel-attention block that pools along height and width, tested as an add-on in DeepLabV3 on VOC and Cityscapes.
• PVT v2: Improved Baselines with Pyramid Vision Transformer: a pyramid transformer with pooled keys and values, overlapping patches and a depthwise convolution in the feed-forward block, tested with Semantic FPN on ADE20K.
• Deep Parametric Continuous Convolutional Neural Networks: a continuous convolution whose kernel is an MLP over relative 3D offsets, for indoor and lidar point clouds.
• SuMa++: Efficient LiDAR-based Semantic SLAM: RangeNet++ labels added to surfel LiDAR mapping, with penalties for moving-object surfels and semantic ICP weights.
• YOLOP: You Only Look Once for Panoptic Driving Perception: one shared encoder with vehicle, drivable-area and lane-line heads, trained jointly.
• Side Adapter Network for Open-Vocabulary Semantic Segmentation: a light side ViT on a frozen CLIP, whose attention biases steer CLIP's last blocks to classify mask proposals.
• nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation: a controlled benchmark of CNN, transformer and Mamba 3D segmenters on six public datasets.
• OMEGA: Efficient Occlusion-Aware Navigation for Air-Ground Robot in Dynamic Environments via State Space Model: OccMamba, a LiDAR network that predicts 3D semantic occupancy with Mamba blocks in separate semantic and geometric branches, plus a planner that avoids building an ESDF map.
• Deep Semantic Segmentation of Natural and Medical Images: A Review: a survey of deep segmentation, with a loss-function section and no experiments of its own.
• RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening: a training-only loss on the style-sensitive feature covariances of a DeepLabV3+.
• Shallow Attention Network for Polyp Segmentation: SANet, a Res2Net decoder with a shallow attention gate, colour exchange in training and logit correction at inference.
• SegDiff: Image Segmentation with Diffusion Probabilistic Models: a conditional diffusion model for masks, trained from scratch, averaging several sampled chains.
• HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation: CNN and Swin pyramids fused by class-token cross-attention, for 2D medical images.
• Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning: U-Net, TernausNet and LinkNet for binary, part and instrument-type masks in surgical video.
• JS3C-Net: a LiDAR segmenter trained with a scene-completion decoder that is dropped at inference.
• A Review on Deep Learning in UAV Remote Sensing: a literature review of deep learning on UAV imagery, with the authors' tallies by network, task, sensor and context.
• A Survey of Deep Learning Techniques for Weed Detection from Images: a survey of 70 deep-learning weed papers and 19 public datasets.
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition: a per-class scale and offset with a learned gate, trained with re-weighted cross-entropy; the segmentation variant is reported on ADE20K.
• Rethinking Semantic Segmentation: A Prototype View: non-learnable cluster-centre prototypes per class, with nearest-prototype prediction.
• A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model: a MaskFormer proposer with frozen CLIP labelling of masked crops against text prompts.
• Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review: a review of vision transformers across medical imaging tasks, with compiled segmentation tables.
• FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation: replaces dilated backbone convolutions with a learned Joint Pyramid Upsampling head.
• Lite-HRNet: A Lightweight High-Resolution Network: a shuffle-block high-resolution network with conditional channel weighting in place of 1x1 convolutions.
• XCiT: Cross-Covariance Image Transformers: attention across feature channels, with cost linear in the number of tokens.
• P2T: Pyramid Pooling Transformer for Scene Understanding: attention keys and values from a pyramid of pooled maps, with depthwise convolutions.
• Omnidata: a rendering pipeline for 21 mid-level labels and a 14.6M-image starter set.
• A ConvNet for the 2020s: ConvNeXt, a pure ConvNet modernised from ResNet-50.
• Image Data Augmentation for Deep Learning: A Survey: a taxonomy of augmentation methods, with segmentation, classification and detection runs for three of its groups.
• Fast Vision Transformers with HiLo Attention: splits attention heads between pooled global attention and windowed local attention; LITv2 is built on it.
• EfficientFormer: Vision Transformers at MobileNet Speed: a pure transformer with convolutional early blocks, sized by an iPhone 12 latency table.
• InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions: a CNN backbone built on DCNv3, a deformable convolution with shared projections and grouped offsets.
• Rethinking Vision Transformers for MobileNet Size and Speed: EfficientFormerV2, a hybrid transformer found by a joint size-speed search.
• RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks: a drop-in convolution with per-window softmax attention over each receptive field.
• PixelLM: Pixel Reasoning with Large Multimodal Model: a multimodal language model that emits one mask per named target through learned codebook tokens.
• Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields: distils LSeg or SAM features into 3D Gaussians for text and prompt queries.
• Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving: LaserMix and LaserMix++, semi-supervised LiDAR segmentation with camera distillation.
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone: a backbone that mixes Mamba and self-attention in its later stages.
• InDRiVE: a driving agent whose input is a segmentation image; not a segmentation method.