--- kind: concept slug: instance-segmentation title: Instance segmentation summary: >- Give each object, map element or 3D scan object its own mask. Copy-Paste pastes masks at no inference cost; BCNet adds an occluder-aware mask head; Mask3D predicts 3D scan masks with Transformer queries; HDMapNet clusters bird's-eye-view embeddings; VisTR predicts per-clip video masks; EQL v2 and DisAlign rebalance rare classes; connected components split buildings. Most need labels. tags: [computer-vision, instance-segmentation, video-segmentation, data-augmentation, autonomous-driving] papers: ["2012.07177", "2107.06307", "2103.12340", "2107.12283", "2012.08548", "2410.17725", "2410.19869", "2012.09856", "2011.14503", "2310.11513", "2210.03105", "2103.16370"] related: [semantic-segmentation, data-augmentation, segmentation-evaluation, bird-eye-view-perception, long-tailed-recognition, class-balanced-loss, self-training, object-detection, graph-neural-network, self-attention, set-prediction, object-queries, bipartite-matching, deep-supervision, text-to-image-generation] status: draft updated_at: 2026-11-01 --- ## Overview Instance segmentation outputs, for every object in an image or a 3D scan, a class label and a mask that separates that object from the others. Training models for it is data-hungry, and mask annotation is slow: the paper cites 22 worker hours per 1000 COCO instance masks [arxiv:2012.07177 §1]. Standard augmentations such as scale jitter are generic and do not treat objects as units [arxiv:2012.07177 §1]. The variants below differ in where the instance comes from. A learned mask head can predict one mask per detected box, with an extra branch for occluders [arxiv:2103.12340 §3.1]. A Transformer decoder can instead predict a fixed set of masks in one pass, with each prediction matched to one ground-truth object, as in the 3D and video variants below [arxiv:2210.03105 §3.1] [arxiv:2011.14503 §3.2]. Mapping work separates road elements by clustering a learned embedding per cell [arxiv:2107.06307 §3.1]. A semantic model can give pixel labels whose connected regions are read as instances [arxiv:2107.12283 §10]. Three papers leave the mask head alone. They change how training is weighted or augmented, or they recalibrate the box classifier: Copy-Paste, EQL v2 and DisAlign [arxiv:2012.07177 §3] [arxiv:2012.08548 §3.2] [arxiv:2103.16370 §3.2]. The same idea appears in mapping for driving. There the units are road elements, such as lane dividers and pedestrian crossings, seen from above. They are separated by clustering a learned embedding per cell rather than by a mask head [arxiv:2107.06307 §3.1]. The invariant the Copy-Paste variant keeps: after a transform, every remaining object still has a correct mask, box and class. Copy-Paste keeps it by building the new labels from the pasted masks and by trimming or dropping the objects it covers [arxiv:2012.07177 §3]. Other instance-segmentation methods, such as the detectors and mask heads the papers build on, have no cards in this wiki yet, so they are not described here. ## Variants ### Copy-Paste augmentation Two random training images are each scale-jittered and flipped independently. A random subset of the instances of one image is pasted onto the other at random positions, and the other image's annotations are updated for occlusion [arxiv:2012.07177 §3]. The paper pastes without a context model: the two prior approaches it cites, Contextual Copy-Paste and InstaBoost, choose paste locations from visual context, and this paper drops that step [arxiv:2012.07177 §1] [arxiv:2012.07177 §2]. - The gains are reported on top of Mask R-CNN-style detectors with FPN, and across backbones from ResNet-50 to EfficientNet-B7 with NAS-FPN, for box AP and mask AP [arxiv:2012.07177 §4.1] [arxiv:2012.07177 tab. 1]. - Large scale jittering on the main image matters most. Switching only the main image back to standard jitter costs about 2 box AP [arxiv:2012.07177 tab. 11]. - Pasting a random subset works better than pasting one object or all of them [arxiv:2012.07177 tab. 10]. - Removing the Gaussian blur from the paste mask gave the same AP, so a hard mask is an acceptable choice [arxiv:2012.07177 tab. 10]. - Self-training can be combined with it: pseudo-labelled images can be backgrounds for pasted ground-truth instances, but pseudo-labelled instances pasted into labelled images did not help [arxiv:2012.07177 §4.4] [arxiv:2012.07177 tab. 3]. Card: [Copy-Paste (Ghiasi et al.)](/papers/2012.07177). ### Instance embeddings with clustering (HDMapNet) HDMapNet predicts a class for each cell of a bird's-eye-view grid, plus an embedding vector for each cell from a separate branch. Inputs are six surrounding cameras, a LiDAR point cloud, or both [arxiv:2107.06307 §3.1]. The training loss pulls the embeddings of one instance towards their mean and pushes the means of different instances apart, with margins of 0.5 and 3.0 [arxiv:2107.06307 §3.1] [arxiv:2107.06307 §4.1]. At inference the paper clusters the embeddings of each class, removes redundant cells, and chains the rest into ordered polylines using a predicted lane direction [arxiv:2107.06307 §3.1]. The paper does not state the clustering parameters. The official code uses a density radius of 1.5 on standardised embeddings and a minimum of 50 cells per cluster [code: postprocess/cluster.py#L159]. - Author-reported on nuScenes, instance mAP over three thresholds is 22.7 for cameras only, 11.6 for LiDAR only and 30.6 for fusion [arxiv:2107.06307 tab. II]. All-class AP at the largest threshold is 34.3, 21.2 and 43.1 for the same three models [arxiv:2107.06307 tab. II]. - Four stacked frames raise camera-only IoU from 32.9 to 36.4. The paper reports this for IoU only, not for instance AP [arxiv:2107.06307 tab. IV]. - The AP threshold units are not stated in the paper. The official code applies thresholds of 2, 4 and 6 metres [code: evaluate_json.py#L10] [code: evaluation/AP.py#L15-L16], so compare these numbers only with results that use the same setup [arxiv:2107.06307 tab. II]. Card: [HDMapNet](/papers/2107.06307). ### Whole-clip sequence prediction for video (VisTR) VisTR runs a CNN and a Transformer encoder-decoder over a clip of 36 frames. It predicts 10 instance sequences per clip in one pass [arxiv:2011.14503 §3.1] [arxiv:2011.14503 §4.1]. Each slot is one object across all frames, so the identity of an instance comes from the output order rather than from a tracker [arxiv:2011.14503 §3.1]. Hungarian matching assigns each sequence to one ground-truth object as a whole, and the mask head uses 3D convolutions over time [arxiv:2011.14503 §3.2] [arxiv:2011.14503 §3.3]. - Author-reported on YouTube-VIS validation, with video mask IoU: 40.1 AP with ResNet-101, at 27.7 FPS with data loading and 57.7 FPS without; 36.2 AP with ResNet-50 [arxiv:2011.14503 tab. 2]. - In a 10-epoch ResNet-50 ablation, a 36-frame clip gives 33.3 AP against 29.7 for an 18-frame clip. Removing the positional encoding gives 28.4 [arxiv:2011.14503 tab. 1]. The 3D convolutions add about 1 AP, 34.4 against 33.3 [arxiv:2011.14503 tab. 1]. - The released inference script reads only the first 36 frames of each video, and the paper's overlap matching across clips is not in that code [arxiv:2011.14503 §4.2] [code: inference.py#L186-L189]. Videos longer than 36 frames need an extra step before they can be used. Card: [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503). ### Transformer queries on 3D scans (Mask3D) Mask3D quantises a 3D scan into voxels and runs a sparse U-Net to get point features at several scales [arxiv:2210.03105 §3]. It keeps 100 instance queries, placed at farthest-point-sampled positions with zero starting features [arxiv:2210.03105 §3.1] [arxiv:2210.03105 app. §1]. A Transformer decoder refines them over several passes, and each query's cross-attention is limited to the voxels inside its mask from the previous layer [arxiv:2210.03105 §3] [arxiv:2210.03105 eq. 3]. Each query yields one class and one binary mask over the whole scene. Hungarian matching pairs each query with at most one ground-truth object, and the mask loss is a BCE plus Dice loss on the matched pairs [arxiv:2210.03105 §3.1] [arxiv:2210.03105 eq. 5]. Every decoder layer gets its own loss [arxiv:2210.03105 eq. 6]. - Author-reported on ScanNet test, mAP is 56.6 against 50.4 for SoftGroup, the best listed baseline [arxiv:2210.03105 tab. 1]. On S3DIS Area 5 the paper reports 56.6 AP against 42.7 for SSTNet [arxiv:2210.03105 tab. 2]. - One 2 cm training run takes about 78 hours on one NVIDIA A40 [arxiv:2210.03105 §3.1]. The runtime in the paper excludes post-processing [arxiv:2210.03105 tab. 1]. - The released code departs from the paper in ways that change results. Its ScanNet validation script uses 150 queries, not 100 [code: scripts/scannet/scannet_val.sh#L6], and its top-k selection ranks query-class pairs rather than queries [code: trainer/trainer.py#L577-L600]. State which setting a reported number uses. Card: [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105). ### Occlusion-aware mask head (BCNet) BCNet adds an occluder branch to the mask head of a detect-then-segment model. The region feature of each box first goes through a graph layer that models the occluding object, which predicts that object's mask and boundary. Its output is added back to the region feature, and a second graph layer then predicts the partly hidden object [arxiv:2103.12340 §3.1] [arxiv:2103.12340 §3.2]. Each graph layer is a non-local, dot-product graph over the pixels of the region [arxiv:2103.12340 §3.2] [arxiv:2103.12340 eq. 1] [arxiv:2103.12340 eq. 2]. - The main experiments use FCOS. A Faster R-CNN variant is also reported [arxiv:2103.12340 §4.2] [arxiv:2103.12340 tab. 4]. - On COCO-OCC, the split of overlap-heavy validation images, modelling the occluder lifts FCOS from 29.04 to 30.37 AP [arxiv:2103.12340 tab. 1]. The bilayer graph raises the single-graph GCN from 29.63 to 30.68 [arxiv:2103.12340 tab. 3]. - With Faster R-CNN and ResNet-101, the paper reports 39.8 mask AP on COCO test-dev, against 37.0 for Mask R-CNN with the same backbone [arxiv:2103.12340 tab. 8]. - It needs more parameters: 54.0M against 51.0M for the FCOS baseline [arxiv:2103.12340 tab. 4]. The released code also runs the occluder heads at test time [code: detectron2/modeling/roi_heads/mask_head.py#L224-L226]. - The paper does not define which annotated object counts as an occluder. The released code reads that mask from an extra per-instance annotation field [arxiv:2103.12340 §3.2] [code: detectron2/data/detection_utils.py#L277]. Card: [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](/papers/2103.12340). ### Connected components on a semantic mask (building footprints) Sirko et al. train a semantic segmenter: a U-Net with a ResNet-50-v2 encoder that gives every pixel a building confidence [arxiv:2107.12283 §4]. Instances are not predicted. The confidence map is thresholded, and the 4-connected components of the result become the buildings. Each instance's score is the mean confidence of its pixels [arxiv:2107.12283 §10]. - Training weights pixels near instance boundaries, through a Gaussian-blurred edge map, so that the network learns to keep neighbouring buildings apart [arxiv:2107.12283 §5.1]. Instance masks are eroded by one pixel before training and dilated again after inference [arxiv:2107.12283 §9] [arxiv:2107.12283 §10]. - Author-reported, mAP at IoU 0.5 on a 1,920-image African test set: 0.601 with ImageNet pre-training against 0.531 with none [arxiv:2107.12283 tab. 3]. Self-training adds +0.057 mAP on average [arxiv:2107.12283 §11] [arxiv:2107.12283 fig. 8]. - The method was run on continental Africa and produced 516M polygons after deduplication [arxiv:2107.12283 §12]. Card: [Continental-Scale Building Detection from High Resolution Satellite Imagery](/papers/2107.12283). ### Gradient-balanced loss for rare classes (EQL v2) EQL v2 is a classification-loss change for a sigmoid detector with one output per category, in the Mask R-CNN style [arxiv:2012.08548 §3.1]. For each class it keeps a running ratio of accumulated positive to negative gradient magnitude. A sigmoid of that ratio sets the weight of negatives, and positives get a boost that grows as negatives are suppressed [arxiv:2012.08548 §3.2] [arxiv:2012.08548 eq. 5] [arxiv:2012.08548 eq. 6]. The paper uses an amplification of 4, a steepness of 12 and a centre of 0.8 [arxiv:2012.08548 §4.2]. - On LVIS v1.0 with a ResNet-50 Mask R-CNN and a 1x schedule, the single-stage method reaches 23.7 AP, 14.9 on rare classes [arxiv:2012.08548 tab. 1]. The same table gives 18.6 for EQL, 22.2 for repeat-factor sampling with 12 epochs and 23.1 for BAGS, a two-stage method with 12+12 epochs [arxiv:2012.08548 tab. 1]. - The method is trained end-to-end with a random sampler and needs no second fine-tuning stage [arxiv:2012.08548 tab. 1] [arxiv:2012.08548 §1]. - The running ratio is never reset. A resumed run inherits the saved state, and changing the number of classes breaks it [arxiv:2012.08548 eq. 9]. Card: [Equalization Loss v2](/papers/2012.08548). ### Classifier recalibration on LVIS (DisAlign) DisAlign starts from a trained Mask R-CNN and keeps its backbone and original classifier fixed. Only a per-class scale and offset, plus a confidence layer that sets, for each input, how much of the adjusted score to use, are trained [arxiv:2103.16370 §3.2] [arxiv:2103.16370 app. C.2]. The class weights are inverse class frequency raised to a power of 0.8, applied to the foreground classes only [arxiv:2103.16370 app. C.2]. The paper does not state the initial value of the learned cosine scale for LVIS; the released config starts it at 20.0 [code: instance_seg/lvis0.5/mask_rcnn/res50/mask_rcnn.res50.fpn.lvis.multiscale.cos_norm.disalign.1x/config.py#L19-L20]. - Author-reported on LVIS v0.5 with a ResNet-50 FPN Mask R-CNN and ImageNet pre-training, mask AP is 24.3 for DisAlign against 21.2 for the joint baseline. The cosine version reaches 26.3 against 23.8 for its baseline. The same table gives 24.3 for repeat-factor sampling and 22.8 for EQL [arxiv:2103.16370 tab. 8]. - With COCO pre-training the cosine version reaches 27.9 mask AP, against 25.3 for its baseline and 26.3 for GroupSoftmax [arxiv:2103.16370 tab. 8]. - On LVIS v1.0 with a Cascade R-CNN and ResNet-50, mask AP rises from 23.5 to 27.0 [arxiv:2103.16370 tab. 9]. The tables give single runs with no seeds or variance [arxiv:2103.16370 §4.3]. - The paper gives a stage-2 length of 9k iterations at learning rate 0.02 [arxiv:2103.16370 app. C.2]. The released config runs 4500 iterations and steps the learning rate at 4000 [code: instance_seg/lvis0.5/mask_rcnn/res50/mask_rcnn.res50.fpn.lvis.multiscale.cos_norm.disalign.1x/config.py#L31-L32]. The loss is a weighted mean in the code, where the paper's loss is a batch mean [code: instance_seg/lvis0.5/mask_rcnn/res50/mask_rcnn.res50.fpn.lvis.multiscale.cos_norm.disalign.1x/roi_heads/fast_rcnn.py#L73-L76]. State which setting a result uses. - The released code loads the class frequencies from `num_shots_v0.5.npy`, which is not in the repository [code: instance_seg/lvis0.5/mask_rcnn/res50/mask_rcnn.res50.fpn.lvis.multiscale.cos_norm.disalign.1x/roi_heads/fast_rcnn.py#L41-L45]. Card: [Distribution Alignment (DisAlign)](/papers/2103.16370). ### Off-the-shelf YOLO-seg models (family comparison) Sapkota and Karkee benchmark YOLO11-seg and YOLOv8-seg, each in five sizes (n, s, m, l, x), on 991 orchard images with two classes: occluded and non-occluded immature green apples [arxiv:2410.19869 §3.2]. This is a comparison, not a new method, and each model was trained once [arxiv:2410.19869 §3.3]. A separate review describes the YOLO11 family, in which YOLO11-seg is one of five task variants, each offered in five sizes [arxiv:2410.17725 §5] [arxiv:2410.17725 tab. 2]. It describes block-level changes, such as C3k2 blocks in place of C2f and a C2PSA attention block after SPPF [arxiv:2410.17725 §4.1.1] [arxiv:2410.17725 §4.1.2]. The review runs no experiments of its own and gives no layer settings, so it cannot be used to check the benchmark numbers below [arxiv:2410.17725 §4] [arxiv:2410.17725 §6]. Card: [YOLOv11: An Overview of the Key Architectural Enhancements](/papers/2410.17725). - Best mask mAP@50 over all classes is YOLO11m-seg at 0.860, against 0.848 for the best YOLOv8 model, YOLOv8l-seg [arxiv:2410.19869 tab. 2]. - On occluded fruit, YOLO11m-seg reaches 0.810 mask mAP@50 and YOLOv8l-seg reaches 0.807 [arxiv:2410.19869 tab. 2]. - YOLOv8n-seg has the lowest inference time, 3.3 ms, against 4.8 ms for YOLO11n-seg. This is the inference column only; the postprocess column of YOLO11n-seg is 56.7 ms [arxiv:2410.19869 tab. 3]. - The smallest models have 2.83M (YOLO11n-seg) and 2.94M (YOLOv8n-seg) parameters, at 10.2 and 10.7 GFLOPs [arxiv:2410.19869 §4.2]. Card: [Comparing YOLOv11 and YOLOv8 for immature green fruit](/papers/2410.19869). ### Mask as an input to another pipeline (hand-object reconstruction) Cao et al. do not output instances. They take a mask from a class-agnostic instance model, trained on COCO, and keep the instance that overlaps the detected hand box most [arxiv:2012.09856 §3.2]. That mask is the target for fitting the object's pose by differentiable rendering [arxiv:2012.09856 §3.2]. Card: [Reconstructing Hand-Object Interactions in the Wild](/papers/2012.09856). ### Instance detector as a scorer (GenEval) GenEval does not produce new instance masks. It runs a Mask2Former with a Swin-S backbone, trained on COCO for instance segmentation, over generated images. Detections above 0.3 are kept, or above 0.9 when the task is counting, and the detections are checked against the object names, counts and relative positions in the prompt [arxiv:2310.11513 §3.2] [arxiv:2310.11513 app. C.3]. The detector is a fixed measuring tool, so the variant is listed here only because the scores depend on it. - Its verdicts agree with human annotators on 83% of images, against 88% between annotators [arxiv:2310.11513 §4]. The counting threshold and the position dead-zone factor were chosen on those same annotations [arxiv:2310.11513 app. A.2]. - The detector covers only the 80 COCO classes. The authors note that it misses fine details and generalises poorly to artistic renders [arxiv:2310.11513 §6] [fig. 4]. Card: [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](/papers/2310.11513). ## When to use which | Situation | Use | Why | |---|---|---| | Instance masks available for the training set | Copy-Paste on top of a Mask R-CNN-style pipeline | Gains on COCO across seven models, about +0.5 to +1.1 mask AP [arxiv:2012.07177 tab. 1] | | Only boxes available | Not applicable as described | The method cuts out and re-masks objects, so it needs masks [arxiv:2012.07177 §3] | | Little labelled data | Copy-Paste with large scale jittering | With 10% of COCO, Copy-Paste adds 6.9 box AP over standard jitter [arxiv:2012.07177 §4.3] | | Rare classes (long-tailed data) | Copy-Paste, with a two-stage classifier for the largest gains | On LVIS, +6.1 mask AP_r for the strongest two-stage model [arxiv:2012.07177 tab. 7] | | Rare classes, sigmoid detector, one training stage | EQL v2 | 23.7 AP and 14.9 AP_r on LVIS v1.0 with 12 epochs, against 18.6 and 2.1 for EQL [arxiv:2012.08548 tab. 1] | | Rare classes on LVIS, a trained Mask R-CNN already exists, and only the classifier should change | DisAlign with rho 0.8 on the box classifier | Mask AP 26.3 against 23.8 for the cosine baseline with ImageNet pre-training [arxiv:2103.16370 tab. 8]; stage 2 runs with the network frozen [arxiv:2103.16370 app. C.2] | | Short training budget | Copy-Paste still helps at 2x/3x schedules | Gains are reported at 24 and 36 epochs [arxiv:2012.07177 §4.2] | | Long training budget | Copy-Paste with large scale jittering, trained to convergence | Converging from scratch takes 576 epochs; the paper reports gains growing with longer training [arxiv:2012.07177 §4.1] [arxiv:2012.07177 §4.2] | | Latency-sensitive inference | Copy-Paste is training-only | It changes only the training images, so it adds no inference cost [arxiv:2012.07177 §3] | | Mixing in mixup | Do not stack it on large scale jittering | Mixup helped only in the low-data regime and not on top of large scale jittering [arxiv:2012.07177 §4.2] [arxiv:2012.07177 §4.3] | | Same-class objects overlap and occluder masks can be labelled | BCNet on FCOS or Faster R-CNN | +2.25 AP on COCO-OCC for FCOS and +2.04 for Faster R-CNN, at a cost of about 3M more parameters for FCOS [arxiv:2103.12340 tab. 4] | | Occluder masks cannot be labelled | BCNet is not applicable as released | The released code reads occluder masks from an extra annotation field [code: detectron2/data/detection_utils.py#L277] | | Single class, semantic labels only, dense or touching objects | Connected components on a semantic mask | 0.601 mAP at IoU 0.5 with ImageNet pre-training; the edge weights target touching objects [arxiv:2107.12283 tab. 3] [arxiv:2107.12283 §5.1] | | Map elements in bird's-eye view, cameras and LiDAR both available | HDMapNet(Fusion) | Highest instance mAP reported, 30.6 against 22.7 for cameras only [arxiv:2107.06307 tab. II] | | Map elements in bird's-eye view, cameras only | HDMapNet(Surr) | The camera-only variant reports 22.7 instance mAP [arxiv:2107.06307 tab. II] | | Map elements in bird's-eye view, LiDAR only | Not recommended on these numbers | Lowest instance mAP of the three, 11.6 [arxiv:2107.06307 tab. II] | | Map elements, more history wanted | Temporal fusion of several frames | Camera-only IoU rises from 32.9 to 36.4 with four frames; no instance AP is reported for this [arxiv:2107.06307 tab. IV] | | Comparing map-element AP with another paper | Check threshold units first | The table's 0.2, 0.5 and 1.0 have no units, and the code uses metres [arxiv:2107.06307 tab. II] | | About 1,000 labelled images, two classes, want a ready model family | YOLO11m-seg for accuracy, YOLOv8n-seg for speed | 0.860 vs 0.848 mask mAP@50 for the best of each family, and 3.3 ms vs 4.8 ms inference for the n sizes [arxiv:2410.19869 tab. 2] [arxiv:2410.19869 tab. 3]; one run per model | | Video, objects must keep one identity across frames, and no tracker is wanted | VisTR, with a clip-matching step added at inference for long videos | 40.1 AP with ResNet-101 on YouTube-VIS validation, author-reported [arxiv:2011.14503 tab. 2]; the released inference code reads only 36 frames [code: inference.py#L186-L189] | | 3D scans (point clouds) with instance labels, one-pass masks wanted | Mask3D, with the paper's settings stated | 56.6 mAP on ScanNet test, author-reported, against 50.4 for SoftGroup [arxiv:2210.03105 tab. 1]; about 78 hours per 2 cm run on one A40 [arxiv:2210.03105 §3.1] | | Score generated images for named objects, counts, colours and positions | GenEval with its COCO detector | Overall 0.61 for IF-XL, 0.55 for SD-XL [arxiv:2310.11513 tab. 2]; a measuring setup, not a new mask method | ## Implementation sketch Language-agnostic outline of Copy-Paste; the other variants' cores are in their cards. Names are ours. Numbers are not fixed by this sketch; the paper's values are in the card. HDMapNet's embedding loss and clustering are described in its variant section above, and are not repeated here because they work on a grid rather than on masks. ```pseudocode # Sample two labelled images, each with masks, boxes and classes [§3] I_main, I_paste = sample(D), sample(D) # Scale-jitter and flip each image on its own [§3] [tab. 11] I_main = jitter_flip(I_main, scale=(0.1, 2.0)) I_paste = jitter_flip(I_paste, scale=(0.1, 2.0)) # Choose a random subset of the pasted image's instances [§3] [tab. 10] S = random_subset(instances(I_paste)) # Paste region is the union of the chosen masks [§3] alpha = union(mask(s) for s in S) # Blend the two images; a Gaussian on alpha is optional [tab. 10] I_new = I_paste * alpha + I_main * (1 - alpha) # Trim the paste region out of the main image's instances [§3] for m in instances(I_main): m = m AND NOT alpha if empty(m): drop m # fully covered: remove label else: box(m) = bbox(m) # partly covered: recompute box # Labels for the new image: trimmed main labels plus pasted labels [§3] labels = main_labels + [(mask(s), box(s), cls(s)) for s in S] ``` ## Contradictions The papers report on different datasets and metrics (COCO, LVIS and Open Images for the Copy-Paste and EQL v2 papers, LVIS v0.5 and v1.0 for DisAlign, nuScenes for HDMapNet, COCO-OCC for BCNet, YouTube-VIS for VisTR, ScanNet, S3DIS and STPLS3D scans for Mask3D, African satellite tiles for the building work, one orchard for the YOLO comparison, and generated images for GenEval), so their numbers are not comparable. The EQL rows of two tables are not comparable. DisAlign reports EQL at 22.8 mask AP on LVIS v0.5 with ImageNet pre-training [arxiv:2103.16370 tab. 8]. EQL v2 reports EQL at 18.6 AP on LVIS v1.0 [arxiv:2012.08548 tab. 1]. The versions and metrics differ, so neither number checks the other. Mask3D's abstract gives a gain of +12.4 mAP on ScanNet200, and its text gives +10.8 [arxiv:2210.03105 §4.1]. The table has no overall mAP to settle this, and the 10.8 matches the gap in head-class AP alone [arxiv:2210.03105 tab. 3]. The paper also gives 40.6 and 40.9 for the same ScanNet validation setting, in two different tables [arxiv:2210.03105 tab. 4] [arxiv:2210.03105 tab. 5]. This page uses the table values. VisTR's authors call it the most accurate single-model method on YouTube-VIS [arxiv:2011.14503 §4.3]. The same table lists MaskProp with ResNet-101 at 42.5 AP, above VisTR's 40.1 [arxiv:2011.14503 tab. 2]. The authors exclude MaskProp because it combines several networks and post-processing [arxiv:2011.14503 §4.3]. This page does not settle the comparison. Within GenEval, the position rule in the paper's appendix differs from the released code, which divides each offset by its length and uses a stricter test [arxiv:2310.11513 app. C.3] [code: evaluation/evaluate_images.py#L145-L156]. Position scores from the code are therefore not the paper's formula. Within the Copy-Paste paper, the COCO training set is given as 118k images [arxiv:2012.07177 §4.1], but the self-training section uses 120k labelled COCO images [arxiv:2012.07177 §4.4]. Unlabelled COCO is also given as 120k in one place [arxiv:2012.07177 §4.1] and as 110k in the pseudo-labelled pool [arxiv:2012.07177 §4.4]. The page uses the 118k figure for the COCO results. BCNet's boundary loss weight is 0.5 in the paper [arxiv:2103.12340 §3.3]. The released code sums two identical binary cross-entropy terms and then applies a 0.5 weight, so the effective weight on one term is 1.0 [code: detectron2/modeling/meta_arch/fcos.py#L208-L209]. A reimplementation that follows the paper's formula trains the boundary head at half that strength. The released code's Faster R-CNN path also calls the mask loss with a different argument count from its definition [code: detectron2/modeling/roi_heads/roi_heads.py#L420] [code: detectron2/modeling/roi_heads/mask_head.py#L24]. The Faster R-CNN numbers therefore cannot be checked against that code without a fix. BCNet's text and table disagree on the FCOS gain on COCO-OCC. The text says 2.23 AP [arxiv:2103.12340 §4.2], while the table gives 30.68 minus 28.43, which is 2.25 [arxiv:2103.12340 tab. 4]. This page uses the table figure. Within the YOLO comparison, the text and the tables disagree in places. The text credits YOLO11n-seg with the best mask precision, but the table gives 0.831 to YOLO11l-seg [arxiv:2410.19869 §4.1] [arxiv:2410.19869 tab. 1]. The text calls YOLO11x-seg the shortest training run, while the training-time figures show a shorter run for YOLO11l-seg [arxiv:2410.19869 §4.3]. This page uses the tables. ## Papers - [Copy-Paste (Ghiasi et al.)](/papers/2012.07177): Copy-Paste augmentation with large scale jittering and self-training; reports COCO and LVIS gains. - [HDMapNet](/papers/2107.06307): instance-embedding branch and clustering for online vector map elements in bird's-eye view, from cameras, LiDAR or both. - [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](/papers/2103.12340): BCNet, an occluder-aware bilayer graph mask head for detect-then-segment models. - [Continental-Scale Building Detection from High Resolution Satellite Imagery](/papers/2107.12283): semantic U-Net with edge weighting and connected components, applied to building footprints across Africa. - [Equalization Loss v2](/papers/2012.08548): gradient-balance classification loss for long-tailed sigmoid detectors, reported on LVIS and Open Images. - [Distribution Alignment (DisAlign)](/papers/2103.16370): second-stage recalibration of a frozen classifier for long-tailed recognition, with LVIS mask results and ADE20K segmentation. - [YOLOv11: An Overview of the Key Architectural Enhancements](/papers/2410.17725): architecture review of the YOLO11 family, including the seg variant; no experiments of its own. - [Comparing YOLOv11 and YOLOv8 for immature green fruit](/papers/2410.19869): benchmark of YOLO11-seg and YOLOv8-seg in five sizes on orchard images. - [Reconstructing Hand-Object Interactions in the Wild](/papers/2012.09856): uses a class-agnostic instance mask as the object input to a hand-object fit. - [End-to-End Video Instance Segmentation with Transformers](/papers/2011.14503): VisTR, whole-clip sequence prediction with Hungarian matching; reports YouTube-VIS results. - [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](/papers/2210.03105): Transformer queries that predict 3D scan instance masks directly, trained with Hungarian matching; reports ScanNet, S3DIS and STPLS3D results. - [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](/papers/2310.11513): text-to-image benchmark that uses a COCO instance detector as its scorer.