Instance segmentation

Overview

Instance segmentation outputs, for every object in an image or a 3D scan, a class label and a mask that separates that object from the others. Training models for it is data-hungry, and mask annotation is slow: the paper cites 22 worker hours per 1000 COCO instance masks 2012.07177 §1. Standard augmentations such as scale jitter are generic and do not treat objects as units 2012.07177 §1.

The variants below differ in where the instance comes from. A learned mask head can predict one mask per detected box, with an extra branch for occluders 2103.12340 §3.1. A Transformer decoder can instead predict a fixed set of masks in one pass, with each prediction matched to one ground-truth object, as in the 3D and video variants below 2210.03105 §3.1 2011.14503 §3.2. Mapping work separates road elements by clustering a learned embedding per cell 2107.06307 §3.1. A semantic model can give pixel labels whose connected regions are read as instances 2107.12283 §10. Three papers leave the mask head alone. They change how training is weighted or augmented, or they recalibrate the box classifier: Copy-Paste, EQL v2 and DisAlign 2012.07177 §3 2012.08548 §3.2 2103.16370 §3.2.

The same idea appears in mapping for driving. There the units are road elements, such as lane dividers and pedestrian crossings, seen from above. They are separated by clustering a learned embedding per cell rather than by a mask head 2107.06307 §3.1.

The invariant the Copy-Paste variant keeps: after a transform, every remaining object still has a correct mask, box and class. Copy-Paste keeps it by building the new labels from the pasted masks and by trimming or dropping the objects it covers 2012.07177 §3.

Other instance-segmentation methods, such as the detectors and mask heads the papers build on, have no cards in this wiki yet, so they are not described here.

Variants

Copy-Paste augmentation

Two random training images are each scale-jittered and flipped independently. A random subset of the instances of one image is pasted onto the other at random positions, and the other image's annotations are updated for occlusion 2012.07177 §3. The paper pastes without a context model: the two prior approaches it cites, Contextual Copy-Paste and InstaBoost, choose paste locations from visual context, and this paper drops that step [arxiv:2012.07177 §1] 2012.07177 §2.

• The gains are reported on top of Mask R-CNN-style detectors with FPN, and across backbones from ResNet-50 to EfficientNet-B7 with NAS-FPN, for box AP and mask AP 2012.07177 §4.1 2012.07177 tab. 1.
• Large scale jittering on the main image matters most. Switching only the main image back to standard jitter costs about 2 box AP [arxiv:2012.07177 tab. 11].
• Pasting a random subset works better than pasting one object or all of them 2012.07177 tab. 10.
• Removing the Gaussian blur from the paste mask gave the same AP, so a hard mask is an acceptable choice 2012.07177 tab. 10.
• Self-training can be combined with it: pseudo-labelled images can be backgrounds for pasted ground-truth instances, but pseudo-labelled instances pasted into labelled images did not help 2012.07177 §4.4 [arxiv:2012.07177 tab. 3].

Card: Copy-Paste (Ghiasi et al.).

Instance embeddings with clustering (HDMapNet)

HDMapNet predicts a class for each cell of a bird's-eye-view grid, plus an embedding vector for each cell from a separate branch. Inputs are six surrounding cameras, a LiDAR point cloud, or both 2107.06307 §3.1. The training loss pulls the embeddings of one instance towards their mean and pushes the means of different instances apart, with margins of 0.5 and 3.0 2107.06307 §3.1 2107.06307 §4.1.

At inference the paper clusters the embeddings of each class, removes redundant cells, and chains the rest into ordered polylines using a predicted lane direction 2107.06307 §3.1. The paper does not state the clustering parameters. The official code uses a density radius of 1.5 on standardised embeddings and a minimum of 50 cells per cluster cluster.py.

• Author-reported on nuScenes, instance mAP over three thresholds is 22.7 for cameras only, 11.6 for LiDAR only and 30.6 for fusion [arxiv:2107.06307 tab. II]. All-class AP at the largest threshold is 34.3, 21.2 and 43.1 for the same three models 2107.06307 tab. II.
• Four stacked frames raise camera-only IoU from 32.9 to 36.4. The paper reports this for IoU only, not for instance AP 2107.06307 tab. IV.
• The AP threshold units are not stated in the paper. The official code applies thresholds of 2, 4 and 6 metres evaluate_json.py AP.py, so compare these numbers only with results that use the same setup 2107.06307 tab. II.

Card: HDMapNet.

Whole-clip sequence prediction for video (VisTR)

VisTR runs a CNN and a Transformer encoder-decoder over a clip of 36 frames. It predicts 10 instance sequences per clip in one pass 2011.14503 §3.1 2011.14503 §4.1. Each slot is one object across all frames, so the identity of an instance comes from the output order rather than from a tracker 2011.14503 §3.1. Hungarian matching assigns each sequence to one ground-truth object as a whole, and the mask head uses 3D convolutions over time 2011.14503 §3.2 2011.14503 §3.3.

• Author-reported on YouTube-VIS validation, with video mask IoU: 40.1 AP with ResNet-101, at 27.7 FPS with data loading and 57.7 FPS without; 36.2 AP with ResNet-50 2011.14503 tab. 2.
• In a 10-epoch ResNet-50 ablation, a 36-frame clip gives 33.3 AP against 29.7 for an 18-frame clip. Removing the positional encoding gives 28.4 2011.14503 tab. 1. The 3D convolutions add about 1 AP, 34.4 against 33.3 2011.14503 tab. 1.
• The released inference script reads only the first 36 frames of each video, and the paper's overlap matching across clips is not in that code 2011.14503 §4.2 inference.py. Videos longer than 36 frames need an extra step before they can be used.

Card: End-to-End Video Instance Segmentation with Transformers.

Transformer queries on 3D scans (Mask3D)

Mask3D quantises a 3D scan into voxels and runs a sparse U-Net to get point features at several scales 2210.03105 §3. It keeps 100 instance queries, placed at farthest-point-sampled positions with zero starting features 2210.03105 §3.1 2210.03105 app. §1. A Transformer decoder refines them over several passes, and each query's cross-attention is limited to the voxels inside its mask from the previous layer 2210.03105 §3 2210.03105 eq. 3. Each query yields one class and one binary mask over the whole scene. Hungarian matching pairs each query with at most one ground-truth object, and the mask loss is a BCE plus Dice loss on the matched pairs 2210.03105 §3.1 2210.03105 eq. 5. Every decoder layer gets its own loss 2210.03105 eq. 6.

• Author-reported on ScanNet test, mAP is 56.6 against 50.4 for SoftGroup, the best listed baseline 2210.03105 tab. 1. On S3DIS Area 5 the paper reports 56.6 AP against 42.7 for SSTNet 2210.03105 tab. 2.
• One 2 cm training run takes about 78 hours on one NVIDIA A40 2210.03105 §3.1. The runtime in the paper excludes post-processing 2210.03105 tab. 1.
• The released code departs from the paper in ways that change results. Its ScanNet validation script uses 150 queries, not 100 [code: scripts/scannet/scannet_val.sh#L6], and its top-k selection ranks query-class pairs rather than queries trainer.py. State which setting a reported number uses.

Card: Mask3D: Mask Transformer for 3D Semantic Instance Segmentation.

Occlusion-aware mask head (BCNet)

BCNet adds an occluder branch to the mask head of a detect-then-segment model. The region feature of each box first goes through a graph layer that models the occluding object, which predicts that object's mask and boundary. Its output is added back to the region feature, and a second graph layer then predicts the partly hidden object 2103.12340 §3.1 2103.12340 §3.2. Each graph layer is a non-local, dot-product graph over the pixels of the region 2103.12340 §3.2 2103.12340 eq. 1 2103.12340 eq. 2.

• The main experiments use FCOS. A Faster R-CNN variant is also reported 2103.12340 §4.2 2103.12340 tab. 4.
• On COCO-OCC, the split of overlap-heavy validation images, modelling the occluder lifts FCOS from 29.04 to 30.37 AP 2103.12340 tab. 1. The bilayer graph raises the single-graph GCN from 29.63 to 30.68 2103.12340 tab. 3.
• With Faster R-CNN and ResNet-101, the paper reports 39.8 mask AP on COCO test-dev, against 37.0 for Mask R-CNN with the same backbone 2103.12340 tab. 8.
• It needs more parameters: 54.0M against 51.0M for the FCOS baseline 2103.12340 tab. 4. The released code also runs the occluder heads at test time mask_head.py.
• The paper does not define which annotated object counts as an occluder. The released code reads that mask from an extra per-instance annotation field 2103.12340 §3.2 detection_utils.py.

Card: Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers.

Connected components on a semantic mask (building footprints)

Sirko et al. train a semantic segmenter: a U-Net with a ResNet-50-v2 encoder that gives every pixel a building confidence 2107.12283 §4. Instances are not predicted. The confidence map is thresholded, and the 4-connected components of the result become the buildings. Each instance's score is the mean confidence of its pixels 2107.12283 §10.

• Training weights pixels near instance boundaries, through a Gaussian-blurred edge map, so that the network learns to keep neighbouring buildings apart 2107.12283 §5.1. Instance masks are eroded by one pixel before training and dilated again after inference 2107.12283 §9 2107.12283 §10.
• Author-reported, mAP at IoU 0.5 on a 1,920-image African test set: 0.601 with ImageNet pre-training against 0.531 with none 2107.12283 tab. 3. Self-training adds +0.057 mAP on average 2107.12283 §11 2107.12283 fig. 8.
• The method was run on continental Africa and produced 516M polygons after deduplication 2107.12283 §12.

Card: Continental-Scale Building Detection from High Resolution Satellite Imagery.

Gradient-balanced loss for rare classes (EQL v2)

EQL v2 is a classification-loss change for a sigmoid detector with one output per category, in the Mask R-CNN style 2012.08548 §3.1. For each class it keeps a running ratio of accumulated positive to negative gradient magnitude. A sigmoid of that ratio sets the weight of negatives, and positives get a boost that grows as negatives are suppressed 2012.08548 §3.2 2012.08548 eq. 5 2012.08548 eq. 6. The paper uses an amplification of 4, a steepness of 12 and a centre of 0.8 2012.08548 §4.2.

• On LVIS v1.0 with a ResNet-50 Mask R-CNN and a 1x schedule, the single-stage method reaches 23.7 AP, 14.9 on rare classes 2012.08548 tab. 1. The same table gives 18.6 for EQL, 22.2 for repeat-factor sampling with 12 epochs and 23.1 for BAGS, a two-stage method with 12+12 epochs 2012.08548 tab. 1.
• The method is trained end-to-end with a random sampler and needs no second fine-tuning stage 2012.08548 tab. 1 2012.08548 §1.
• The running ratio is never reset. A resumed run inherits the saved state, and changing the number of classes breaks it 2012.08548 eq. 9.

Card: Equalization Loss v2.

Classifier recalibration on LVIS (DisAlign)

DisAlign starts from a trained Mask R-CNN and keeps its backbone and original classifier fixed. Only a per-class scale and offset, plus a confidence layer that sets, for each input, how much of the adjusted score to use, are trained 2103.16370 §3.2 2103.16370 app. C.2. The class weights are inverse class frequency raised to a power of 0.8, applied to the foreground classes only 2103.16370 app. C.2. The paper does not state the initial value of the learned cosine scale for LVIS; the released config starts it at 20.0 config.py.

• Author-reported on LVIS v0.5 with a ResNet-50 FPN Mask R-CNN and ImageNet pre-training, mask AP is 24.3 for DisAlign against 21.2 for the joint baseline. The cosine version reaches 26.3 against 23.8 for its baseline. The same table gives 24.3 for repeat-factor sampling and 22.8 for EQL 2103.16370 tab. 8.
• With COCO pre-training the cosine version reaches 27.9 mask AP, against 25.3 for its baseline and 26.3 for GroupSoftmax 2103.16370 tab. 8.
• On LVIS v1.0 with a Cascade R-CNN and ResNet-50, mask AP rises from 23.5 to 27.0 2103.16370 tab. 9. The tables give single runs with no seeds or variance 2103.16370 §4.3.
• The paper gives a stage-2 length of 9k iterations at learning rate 0.02 2103.16370 app. C.2. The released config runs 4500 iterations and steps the learning rate at 4000 config.py. The loss is a weighted mean in the code, where the paper's loss is a batch mean fast_rcnn.py. State which setting a result uses.
• The released code loads the class frequencies from num_shots_v0.5.npy, which is not in the repository fast_rcnn.py.

Card: Distribution Alignment (DisAlign).

Off-the-shelf YOLO-seg models (family comparison)

Sapkota and Karkee benchmark YOLO11-seg and YOLOv8-seg, each in five sizes (n, s, m, l, x), on 991 orchard images with two classes: occluded and non-occluded immature green apples 2410.19869 §3.2. This is a comparison, not a new method, and each model was trained once 2410.19869 §3.3.

A separate review describes the YOLO11 family, in which YOLO11-seg is one of five task variants, each offered in five sizes 2410.17725 §5 2410.17725 tab. 2. It describes block-level changes, such as C3k2 blocks in place of C2f and a C2PSA attention block after SPPF 2410.17725 §4.1.1 2410.17725 §4.1.2. The review runs no experiments of its own and gives no layer settings, so it cannot be used to check the benchmark numbers below 2410.17725 §4 2410.17725 §6. Card: YOLOv11: An Overview of the Key Architectural Enhancements.

• Best mask mAP@50 over all classes is YOLO11m-seg at 0.860, against 0.848 for the best YOLOv8 model, YOLOv8l-seg 2410.19869 tab. 2.
• On occluded fruit, YOLO11m-seg reaches 0.810 mask mAP@50 and YOLOv8l-seg reaches 0.807 2410.19869 tab. 2.
• YOLOv8n-seg has the lowest inference time, 3.3 ms, against 4.8 ms for YOLO11n-seg. This is the inference column only; the postprocess column of YOLO11n-seg is 56.7 ms 2410.19869 tab. 3.
• The smallest models have 2.83M (YOLO11n-seg) and 2.94M (YOLOv8n-seg) parameters, at 10.2 and 10.7 GFLOPs 2410.19869 §4.2.

Card: Comparing YOLOv11 and YOLOv8 for immature green fruit.

Mask as an input to another pipeline (hand-object reconstruction)

Cao et al. do not output instances. They take a mask from a class-agnostic instance model, trained on COCO, and keep the instance that overlaps the detected hand box most 2012.09856 §3.2. That mask is the target for fitting the object's pose by differentiable rendering 2012.09856 §3.2.

Card: Reconstructing Hand-Object Interactions in the Wild.

Instance detector as a scorer (GenEval)

GenEval does not produce new instance masks. It runs a Mask2Former with a Swin-S backbone, trained on COCO for instance segmentation, over generated images. Detections above 0.3 are kept, or above 0.9 when the task is counting, and the detections are checked against the object names, counts and relative positions in the prompt 2310.11513 §3.2 2310.11513 app. C.3. The detector is a fixed measuring tool, so the variant is listed here only because the scores depend on it.

• Its verdicts agree with human annotators on 83% of images, against 88% between annotators 2310.11513 §4. The counting threshold and the position dead-zone factor were chosen on those same annotations 2310.11513 app. A.2.
• The detector covers only the 80 COCO classes. The authors note that it misses fine details and generalises poorly to artistic renders 2310.11513 §6 fig. 4.

Card: GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.

When to use which

Implementation sketch

Language-agnostic outline of Copy-Paste; the other variants' cores are in their cards. Names are ours. Numbers are not fixed by this sketch; the paper's values are in the card. HDMapNet's embedding loss and clustering are described in its variant section above, and are not repeated here because they work on a grid rather than on masks.

# Sample two labelled images, each with masks, boxes and classes [§3]
I_main, I_paste = sample(D), sample(D)
# Scale-jitter and flip each image on its own [§3] [tab. 11]
I_main = jitter_flip(I_main, scale=(0.1, 2.0))
I_paste = jitter_flip(I_paste, scale=(0.1, 2.0))
# Choose a random subset of the pasted image's instances [§3] [tab. 10]
S = random_subset(instances(I_paste))
# Paste region is the union of the chosen masks [§3]
alpha = union(mask(s) for s in S)
# Blend the two images; a Gaussian on alpha is optional [tab. 10]
I_new = I_paste * alpha + I_main * (1 - alpha)
# Trim the paste region out of the main image's instances [§3]
for m in instances(I_main):
    m = m AND NOT alpha
    if empty(m): drop m          # fully covered: remove label
    else: box(m) = bbox(m)       # partly covered: recompute box
# Labels for the new image: trimmed main labels plus pasted labels [§3]
labels = main_labels + [(mask(s), box(s), cls(s)) for s in S]

Contradictions

The papers report on different datasets and metrics (COCO, LVIS and Open Images for the Copy-Paste and EQL v2 papers, LVIS v0.5 and v1.0 for DisAlign, nuScenes for HDMapNet, COCO-OCC for BCNet, YouTube-VIS for VisTR, ScanNet, S3DIS and STPLS3D scans for Mask3D, African satellite tiles for the building work, one orchard for the YOLO comparison, and generated images for GenEval), so their numbers are not comparable.

The EQL rows of two tables are not comparable. DisAlign reports EQL at 22.8 mask AP on LVIS v0.5 with ImageNet pre-training 2103.16370 tab. 8. EQL v2 reports EQL at 18.6 AP on LVIS v1.0 2012.08548 tab. 1. The versions and metrics differ, so neither number checks the other.

Mask3D's abstract gives a gain of +12.4 mAP on ScanNet200, and its text gives +10.8 2210.03105 §4.1. The table has no overall mAP to settle this, and the 10.8 matches the gap in head-class AP alone 2210.03105 tab. 3. The paper also gives 40.6 and 40.9 for the same ScanNet validation setting, in two different tables 2210.03105 tab. 4 2210.03105 tab. 5. This page uses the table values.

VisTR's authors call it the most accurate single-model method on YouTube-VIS 2011.14503 §4.3. The same table lists MaskProp with ResNet-101 at 42.5 AP, above VisTR's 40.1 2011.14503 tab. 2. The authors exclude MaskProp because it combines several networks and post-processing 2011.14503 §4.3. This page does not settle the comparison.

Within GenEval, the position rule in the paper's appendix differs from the released code, which divides each offset by its length and uses a stricter test 2310.11513 app. C.3 evaluate_images.py. Position scores from the code are therefore not the paper's formula.

Within the Copy-Paste paper, the COCO training set is given as 118k images 2012.07177 §4.1, but the self-training section uses 120k labelled COCO images 2012.07177 §4.4. Unlabelled COCO is also given as 120k in one place 2012.07177 §4.1 and as 110k in the pseudo-labelled pool 2012.07177 §4.4. The page uses the 118k figure for the COCO results.

BCNet's boundary loss weight is 0.5 in the paper 2103.12340 §3.3. The released code sums two identical binary cross-entropy terms and then applies a 0.5 weight, so the effective weight on one term is 1.0 [code: detectron2/modeling/meta_arch/fcos.py#L208-L209]. A reimplementation that follows the paper's formula trains the boundary head at half that strength. The released code's Faster R-CNN path also calls the mask loss with a different argument count from its definition roi_heads.py mask_head.py. The Faster R-CNN numbers therefore cannot be checked against that code without a fix.

BCNet's text and table disagree on the FCOS gain on COCO-OCC. The text says 2.23 AP 2103.12340 §4.2, while the table gives 30.68 minus 28.43, which is 2.25 2103.12340 tab. 4. This page uses the table figure.

Within the YOLO comparison, the text and the tables disagree in places. The text credits YOLO11n-seg with the best mask precision, but the table gives 0.831 to YOLO11l-seg 2410.19869 §4.1 2410.19869 tab. 1. The text calls YOLO11x-seg the shortest training run, while the training-time figures show a shorter run for YOLO11l-seg 2410.19869 §4.3. This page uses the tables.

Papers

• Copy-Paste (Ghiasi et al.): Copy-Paste augmentation with large scale jittering and self-training; reports COCO and LVIS gains.
• HDMapNet: instance-embedding branch and clustering for online vector map elements in bird's-eye view, from cameras, LiDAR or both.
• Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers: BCNet, an occluder-aware bilayer graph mask head for detect-then-segment models.
• Continental-Scale Building Detection from High Resolution Satellite Imagery: semantic U-Net with edge weighting and connected components, applied to building footprints across Africa.
• Equalization Loss v2: gradient-balance classification loss for long-tailed sigmoid detectors, reported on LVIS and Open Images.
• Distribution Alignment (DisAlign): second-stage recalibration of a frozen classifier for long-tailed recognition, with LVIS mask results and ADE20K segmentation.
• YOLOv11: An Overview of the Key Architectural Enhancements: architecture review of the YOLO11 family, including the seg variant; no experiments of its own.
• Comparing YOLOv11 and YOLOv8 for immature green fruit: benchmark of YOLO11-seg and YOLOv8-seg in five sizes on orchard images.
• Reconstructing Hand-Object Interactions in the Wild: uses a class-agnostic instance mask as the object input to a hand-object fit.
• End-to-End Video Instance Segmentation with Transformers: VisTR, whole-clip sequence prediction with Hungarian matching; reports YouTube-VIS results.
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation: Transformer queries that predict 3D scan instance masks directly, trained with Hungarian matching; reports ScanNet, S3DIS and STPLS3D results.
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment: text-to-image benchmark that uses a COCO instance detector as its scorer.