--- kind: concept slug: panoptic-segmentation title: Panoptic segmentation summary: >- Every pixel gets a class, and countable objects also get an instance ID. Image models predict one mask per query and resolve overlaps at inference, by pixel argmax or by mask-wise merging. Neural-field models fuse posed views into one 3D scene so IDs agree across views. Trade-off: per-image speed and reproducibility versus multi-view consistency and offline cost. tags: [segmentation, panoptic-segmentation, transformer, neural-rendering, 3d-scene-understanding] papers: ["2109.03814", "2005.12872", "2205.04334", "2212.09802"] related: [instance-segmentation, semantic-segmentation, segmentation-evaluation, set-prediction, bipartite-matching, deep-supervision, object-detection, neural-radiance-field, test-time-augmentation] status: draft updated_at: 2026-11-09 --- ## Overview Panoptic segmentation gives every pixel one label. Stuff such as sky or road gets a class only. Countable things such as people or cars also get an instance ID, so two cars in one image stay separate [arxiv:2109.03814 §1]. The output is therefore one class map and one ID map, and each pixel belongs to exactly one segment. The variants differ in what produces the masks. Four papers are covered here. Two work on a single image. Each predicts a fixed set of queries, and every query yields one class and one mask. Overlaps between masks must then be removed at inference, which is where the papers differ most [arxiv:2109.03814 §3.3.3] [arxiv:2005.12872 §4.4]. The other two build a neural field from several posed views. The field renders one panoptic labelling for any camera, and the IDs agree across views [arxiv:2212.09802 §3.2] [arxiv:2205.04334 §3.1]. The invariant all variants keep is that, after inference, each pixel has one class and at most one instance ID. How they reach it is the design choice. ## Variants ### Separate query sets with mask-wise merging (Panoptic SegFormer) The model uses two query sets. It has 300 thing queries, matched one-to-one with ground-truth objects by Hungarian matching. It has one stuff query per stuff class, so stuff needs no matching [arxiv:2109.03814 §3.1] [arxiv:2109.03814 §3.4] [arxiv:2109.03814 app. A]. A location decoder refines only the thing queries. An auxiliary box and class head supervises it in training and is dropped at inference [arxiv:2109.03814 §3.3.2]. Masks come from the attention maps of the mask decoder. The maps of the three feature levels are brought to one resolution and combined per query. Every decoder layer gets the mask loss [arxiv:2109.03814 §3.3.3] [arxiv:2109.03814 eq. 1] [arxiv:2109.03814 eq. 2]. At inference, each mask gets a score from its class probability and its mask quality. Masks are taken in score order, and each keeps only the pixels that no earlier mask has claimed [arxiv:2109.03814 eq. 6] [arxiv:2109.03814 alg. 1]. The authors set a score threshold of 0.25 for things and an overlap threshold of 0.6 [arxiv:2109.03814 app. A]. - Author-reported on COCO val with a ResNet-50 backbone, 49.6 PQ after 24 epochs and 48.0 after 12 [arxiv:2109.03814 tab. 1]. The MaskFormer baseline reaches 46.5 after 300 epochs [arxiv:2109.03814 tab. 1]. With Swin-L it reaches 55.8 PQ on val and 56.2 on test-dev [arxiv:2109.03814 tab. 1] [arxiv:2109.03814 tab. 2]. - Separate query sets matter. Joint matching gives 48.5 PQ, against 49.6 for decoupled queries, and stuff PQ moves from 39.5 to 42.4 [arxiv:2109.03814 §4.3]. - On a ResNet-50 model, the authors measure 7.8 FPS on a V100, against 4.9 for the DETR baseline [arxiv:2109.03814 tab. 5]. - The code is Apache-2.0 and the paper is CC BY 4.0. The released code departs from the paper in several places, such as where thing classes come from and the stuff overlap rule. Use the paper's numbers only with the code's settings stated [arxiv:2109.03814 §3.3.3] [code: easymd/models/panformer/panformer_head.py#L980-L984]. Card: [Panoptic SegFormer](/papers/2109.03814). ### DETR with a mask head on attention maps (DETR panoptic) DETR is first trained for boxes, for things and stuff alike. Boxes are needed because matching is computed from box distances [arxiv:2005.12872 §4.4]. A mask head then takes the multi-head attention scores over the encoder output and upsamples them with an FPN-like network to stride-4 masks [arxiv:2005.12872 §4.4] [arxiv:2005.12872 fig. 8]. Each mask is supervised with a Dice loss and a focal loss [arxiv:2005.12872 eq. 11]. Training runs in two steps. The boxes are frozen, and only the mask head is trained, for 25 epochs. The authors report that joint training gives similar results [arxiv:2005.12872 §4.4]. At inference, detections below a confidence of 0.85 are dropped. The masks are then combined by a per-pixel argmax, which guarantees they do not overlap. Masks of the same stuff class are merged, and masks under four pixels are removed [arxiv:2005.12872 §4.4]. - Author-reported on COCO val, R50 gives 43.4 PQ, DC5 gives 44.6 and R101 gives 45.1 [arxiv:2005.12872 tab. 5]. The test-set figure is 46 PQ [arxiv:2005.12872 §4.4]. Most of the gain is on stuff, and mask AP on things trails the baselines by up to 8 points [arxiv:2005.12872 §4.4]. - The head is under-specified. The paper gives the Dice loss, but not the focal-loss weight or the number of attention heads M [arxiv:2005.12872 §4.4] [arxiv:2005.12872 app. A.2]. A reproduction has to choose both. - The card has no code check for this paper, so the code licence is not recorded. The paper is CC0 1.0. Card: [End-to-End Object Detection with Transformers](/papers/2005.12872). ### Per-object neural fields in tracked boxes (Panoptic Neural Fields) Panoptic Neural Fields split a driving scene into a background model and one model per tracked object. The background model covers a large box and has a semantic head that ignores the view direction, so labels do not change with the viewing angle. Each object model lives inside an oriented 3D box, whose rotation and translation change over time while its size stays fixed [arxiv:2205.04334 §3.1]. At each point the field combines the boxes it falls in, and the background is used where no box is hit [arxiv:2205.04334 eq. 1]. Training renders colour and semantic logits along rays, and it fits them to the input images and to predicted 2D segmentations. The boxes and tracks are optimised together with the networks, because even the detector's boxes are noisy [arxiv:2205.04334 §3.3]. Inputs come from other tools: an RGB 3D detector, a 3D tracker, SLAM poses and a 2D panoptic network [arxiv:2205.04334 §1] [arxiv:2205.04334 §3.1]. Object MLPs can start from a prior learned on car shapes with FedAvg, which the authors say speeds up convergence [arxiv:2205.04334 §3.4]. - Author-reported on KITTI, the PQ on rendered views is 45.9. Panoptic-DeepLab reaches 43.2 PQ on ground-truth images and 24.9 PQ on NeRF-rendered images [arxiv:2205.04334 tab. 4]. These baselines use different inputs, so the comparison is not like-for-like. - The paper gives no official code, and the card found no release [arxiv:2205.04334 app. C]. - Box membership is not written out. The paper sums the object outputs over all objects, but the text says only box hits count, and the output outside a box is not defined [arxiv:2205.04334 §3.1] [arxiv:2205.04334 eq. 1]. A reproduction must pick a rule and state it. Card: [Panoptic Neural Fields](/papers/2205.04334). ### Neural field that fuses noisy 2D panoptic labels (Panoptic Lifting) Panoptic Lifting trains one TensoRF-style field on posed RGB images and on 2D panoptic masks from a pretrained labeller. The labels come from Mask2Former run on several transformed copies of each image, and the outputs are fused into one panoptic map with a per-pixel confidence [arxiv:2212.09802 §4] [arxiv:2212.09802 app. TTA]. No 3D labels are used [arxiv:2212.09802 §4]. The field renders two probability distributions along each ray, one over classes and one over surrogate instance IDs [arxiv:2212.09802 §3.2]. Each image's 2D instances are matched to surrogate IDs by a linear assignment, which keeps one ID per instance across views [arxiv:2212.09802 eq. 5] [arxiv:2212.09802 eq. 6]. A segment consistency term asks the rays of one 2D segment to agree on a class [arxiv:2212.09802 eq. 7]. Gradients from the label losses are blocked before the density, so label noise cannot reshape the geometry [arxiv:2212.09802 §3.3]. - Author-reported on ScanNet, held-out views, 65.2 mIoU and 58.9 PQ-scene. The labeller alone gets 46.7 mIoU [arxiv:2212.09802 tab. 1]. HyperSim gives 60.1 PQ-scene [arxiv:2212.09802 tab. 1]. - Test-time augmentation is load-bearing. Without it, ScanNet falls to 63.1 mIoU and 55.2 PQ-scene. Removing the gradient blocking gives 61.6 and 53.7 [arxiv:2212.09802 tab. 2]. - Cost: 450k iterations, about 10 hours on one A6000, and the pipeline needs offline pose estimation and label generation [arxiv:2212.09802 §3.3] [arxiv:2212.09802 §4.1.3]. - The released code is CC BY-NC-SA 4.0, and the augmentation code is in a separate fork that is not in the repository [code: README.md#L106]. Use the paper for the method and re-implement the augmentation step. - The labeller limits the label set to the classes it was trained on [arxiv:2212.09802 §4.1.3]. Card: [Panoptic Lifting](/papers/2212.09802). ## When to use which | Situation | Use | Why | |---|---|---| | One RGB image, things and stuff, COCO-style labels, best PQ per training budget | Panoptic SegFormer with ResNet-50 | 49.6 PQ after 24 epochs, against 43.4 for the DETR panoptic head [arxiv:2109.03814 tab. 1] [arxiv:2005.12872 tab. 5]; author-reported | | A DETR-type detector already exists and only a mask head is to be added | DETR panoptic head, trained in two steps | Boxes stay frozen and only the mask head is trained for 25 epochs [arxiv:2005.12872 §4.4]; the head is under-specified, so expect to choose its missing values | | Pixel-level output at a deployed speed matters | Panoptic SegFormer over the DETR baseline | 7.8 FPS against 4.9 on a V100 for ResNet-50 [arxiv:2109.03814 tab. 5]; both are author-reported | | Posed images of one static scene, IDs must agree across views, labels are noisy per frame | Panoptic Lifting | 65.2 mIoU and 58.9 PQ-scene on ScanNet, against 46.7 mIoU for the labeller [arxiv:2212.09802 tab. 1]; needs about 10 hours per scene on an A6000 [arxiv:2212.09802 §3.3] | | Moving rigid objects in driving scenes, per-object editing wanted | Panoptic Neural Fields | Each object gets its own model in a tracked box [arxiv:2205.04334 §3.1]; needs tracks and boxes from other tools | | Code will be reused in a commercial product | Panoptic SegFormer code (Apache-2.0), or re-implement from the DETR and Panoptic Lifting papers | Panoptic Lifting code is non-commercial [code: README.md#L106]; the DETR code was not checked | | Scores must be compared with published numbers | Use the same metric variant | Per-image COCO PQ and scene-level PQ are different measures (see Contradictions) | ## Implementation sketch The shared core of the two image models is the step that turns per-query masks into one panoptic map. Names are ours. The mask-wise rule of Panoptic SegFormer is the more involved one; the DETR rule is a pixel argmax. ```pseudocode # Panoptic map from N query masks, mask-wise rule [arxiv:2109.03814 alg. 1] # Input: cls (N,) class ids, probs (N,), mask_logits (N, H, W), is_thing (N,) # Output: class map (H, W), instance-id map (H, W) def merge_masks(cls, probs, mask_logits, is_thing, t_cnf=0.25, t_keep=0.6): # Binarise each mask at 0.5 [arxiv:2109.03814 alg. 1] binary = mask_logits > 0.5 # Quality: mean of the positive values; the authors leave the average # undefined, and the code adds 1 to the count [arxiv:2109.03814 eq. 6] quality = (binary * mask_logits).sum((1, 2)) / (binary.sum((1, 2)) + 1) # Score: class probability times squared quality [arxiv:2109.03814 eq. 6] score = probs * quality ** 2 cls_map = zeros(H, W); inst_map = zeros(H, W); next_id = 1 # Visit masks from most to least confident [arxiv:2109.03814 alg. 1] for i in argsort(-score): if score[i] < t_cnf: continue # Keep only pixels no earlier mask has claimed [arxiv:2109.03814 alg. 1] kept = binary[i] & (cls_map == 0) if kept.sum() < t_keep * binary[i].sum(): continue cls_map[kept] = cls[i] if is_thing[i]: inst_map[kept] = next_id; next_id += 1 return cls_map, inst_map ``` ```pseudocode # Pixel-wise alternative used by DETR [arxiv:2005.12872 §4.4] # Each pixel takes the mask with the highest score; no overlaps remain. label_map = argmax(mask_scores, axis=query) # (H, W) ``` ## Contradictions - **Panoptic FPN baseline.** The Panoptic SegFormer table lists Panoptic FPN with ResNet-50 at 41.5 PQ after 36 epochs [arxiv:2109.03814 tab. 1]. The DETR table lists a retrained PanopticFPN++ at 42.4 PQ, with DETR's augmentation and an 18x schedule [arxiv:2005.12872 tab. 5]. The two rows use different training recipes, so the gap is not a result about the architecture. - **Merge rule for the same DETR model.** The DETR paper reports 43.4 PQ with a pixel-wise argmax [arxiv:2005.12872 §4.4] [arxiv:2005.12872 tab. 5]. Panoptic SegFormer reports 44.7 for that model with its mask-wise merge [arxiv:2109.03814 §4.3]. A DETR number should name the merge rule. - **DETR epochs.** The Panoptic SegFormer main table and ablation list DETR at 325 epochs, but its appendix lists 500 [arxiv:2109.03814 tab. 1] [arxiv:2109.03814 app. B]. - **PQ is not one quantity.** The COCO results are per-image PQ on single images [arxiv:2109.03814 §4.2]. Panoptic Lifting reports scene-level PQ, computed by tiling each scene's predictions and ground truth [arxiv:2212.09802 §4]. Panoptic Neural Fields report PQ on rendered views of KITTI [arxiv:2205.04334 tab. 4]. A 58.9 PQ-scene and a 49.6 COCO PQ are not comparable numbers. - **Stuff and things by query.** Panoptic SegFormer says that boxes are a poor representation for stuff [arxiv:2109.03814 §1]. The DETR panoptic extension trains boxes for stuff too, because its matching depends on boxes [arxiv:2005.12872 §4.4]. The two designs answer the question differently. ## Papers - [Panoptic SegFormer](/papers/2109.03814): separate thing and stuff queries, a deeply supervised mask decoder and mask-wise merging; reports COCO and ADE20K PQ. - [End-to-End Object Detection with Transformers](/papers/2005.12872): DETR, with a two-step mask head for its panoptic extension and a pixel-wise argmax at inference. - [Panoptic Neural Fields](/papers/2205.04334): per-object neural fields in tracked 3D boxes with a background model, trained on driving scenes. - [Panoptic Lifting](/papers/2212.09802): one neural field fitted to noisy 2D panoptic masks, with test-time augmentation and an injective ID assignment across views.