Panoptic segmentation

Overview

Panoptic segmentation gives every pixel one label. Stuff such as sky or road gets a class only. Countable things such as people or cars also get an instance ID, so two cars in one image stay separate 2109.03814 §1. The output is therefore one class map and one ID map, and each pixel belongs to exactly one segment.

The variants differ in what produces the masks. Four papers are covered here. Two work on a single image. Each predicts a fixed set of queries, and every query yields one class and one mask. Overlaps between masks must then be removed at inference, which is where the papers differ most [arxiv:2109.03814 §3.3.3] 2005.12872 §4.4. The other two build a neural field from several posed views. The field renders one panoptic labelling for any camera, and the IDs agree across views 2212.09802 §3.2 2205.04334 §3.1.

The invariant all variants keep is that, after inference, each pixel has one class and at most one instance ID. How they reach it is the design choice.

Variants

Separate query sets with mask-wise merging (Panoptic SegFormer)

The model uses two query sets. It has 300 thing queries, matched one-to-one with ground-truth objects by Hungarian matching. It has one stuff query per stuff class, so stuff needs no matching 2109.03814 §3.1 [arxiv:2109.03814 §3.4] 2109.03814 app. A. A location decoder refines only the thing queries. An auxiliary box and class head supervises it in training and is dropped at inference 2109.03814 §3.3.2.

Masks come from the attention maps of the mask decoder. The maps of the three feature levels are brought to one resolution and combined per query. Every decoder layer gets the mask loss 2109.03814 §3.3.3 [arxiv:2109.03814 eq. 1] 2109.03814 eq. 2.

At inference, each mask gets a score from its class probability and its mask quality. Masks are taken in score order, and each keeps only the pixels that no earlier mask has claimed 2109.03814 eq. 6 2109.03814 alg. 1. The authors set a score threshold of 0.25 for things and an overlap threshold of 0.6 2109.03814 app. A.

• Author-reported on COCO val with a ResNet-50 backbone, 49.6 PQ after 24 epochs and 48.0 after 12 2109.03814 tab. 1. The MaskFormer baseline reaches 46.5 after 300 epochs 2109.03814 tab. 1. With Swin-L it reaches 55.8 PQ on val and 56.2 on test-dev 2109.03814 tab. 1 2109.03814 tab. 2.
• Separate query sets matter. Joint matching gives 48.5 PQ, against 49.6 for decoupled queries, and stuff PQ moves from 39.5 to 42.4 [arxiv:2109.03814 §4.3].
• On a ResNet-50 model, the authors measure 7.8 FPS on a V100, against 4.9 for the DETR baseline 2109.03814 tab. 5.
• The code is Apache-2.0 and the paper is CC BY 4.0. The released code departs from the paper in several places, such as where thing classes come from and the stuff overlap rule. Use the paper's numbers only with the code's settings stated 2109.03814 §3.3.3 panformer_head.py.

Card: Panoptic SegFormer.

DETR with a mask head on attention maps (DETR panoptic)

DETR is first trained for boxes, for things and stuff alike. Boxes are needed because matching is computed from box distances 2005.12872 §4.4. A mask head then takes the multi-head attention scores over the encoder output and upsamples them with an FPN-like network to stride-4 masks [arxiv:2005.12872 §4.4] 2005.12872 fig. 8. Each mask is supervised with a Dice loss and a focal loss 2005.12872 eq. 11.

Training runs in two steps. The boxes are frozen, and only the mask head is trained, for 25 epochs. The authors report that joint training gives similar results 2005.12872 §4.4.

At inference, detections below a confidence of 0.85 are dropped. The masks are then combined by a per-pixel argmax, which guarantees they do not overlap. Masks of the same stuff class are merged, and masks under four pixels are removed 2005.12872 §4.4.

• Author-reported on COCO val, R50 gives 43.4 PQ, DC5 gives 44.6 and R101 gives 45.1 2005.12872 tab. 5. The test-set figure is 46 PQ [arxiv:2005.12872 §4.4]. Most of the gain is on stuff, and mask AP on things trails the baselines by up to 8 points 2005.12872 §4.4.
• The head is under-specified. The paper gives the Dice loss, but not the focal-loss weight or the number of attention heads M 2005.12872 §4.4 2005.12872 app. A.2. A reproduction has to choose both.
• The card has no code check for this paper, so the code licence is not recorded. The paper is CC0 1.0.

Card: End-to-End Object Detection with Transformers.

Per-object neural fields in tracked boxes (Panoptic Neural Fields)

Panoptic Neural Fields split a driving scene into a background model and one model per tracked object. The background model covers a large box and has a semantic head that ignores the view direction, so labels do not change with the viewing angle. Each object model lives inside an oriented 3D box, whose rotation and translation change over time while its size stays fixed [arxiv:2205.04334 §3.1]. At each point the field combines the boxes it falls in, and the background is used where no box is hit 2205.04334 eq. 1.

Training renders colour and semantic logits along rays, and it fits them to the input images and to predicted 2D segmentations. The boxes and tracks are optimised together with the networks, because even the detector's boxes are noisy 2205.04334 §3.3.

Inputs come from other tools: an RGB 3D detector, a 3D tracker, SLAM poses and a 2D panoptic network 2205.04334 §1 2205.04334 §3.1. Object MLPs can start from a prior learned on car shapes with FedAvg, which the authors say speeds up convergence 2205.04334 §3.4.

• Author-reported on KITTI, the PQ on rendered views is 45.9. Panoptic-DeepLab reaches 43.2 PQ on ground-truth images and 24.9 PQ on NeRF-rendered images 2205.04334 tab. 4. These baselines use different inputs, so the comparison is not like-for-like.
• The paper gives no official code, and the card found no release 2205.04334 app. C.
• Box membership is not written out. The paper sums the object outputs over all objects, but the text says only box hits count, and the output outside a box is not defined 2205.04334 §3.1 2205.04334 eq. 1. A reproduction must pick a rule and state it.

Card: Panoptic Neural Fields.

Neural field that fuses noisy 2D panoptic labels (Panoptic Lifting)

Panoptic Lifting trains one TensoRF-style field on posed RGB images and on 2D panoptic masks from a pretrained labeller. The labels come from Mask2Former run on several transformed copies of each image, and the outputs are fused into one panoptic map with a per-pixel confidence 2212.09802 §4 2212.09802 app. TTA. No 3D labels are used 2212.09802 §4.

The field renders two probability distributions along each ray, one over classes and one over surrogate instance IDs 2212.09802 §3.2. Each image's 2D instances are matched to surrogate IDs by a linear assignment, which keeps one ID per instance across views 2212.09802 eq. 5 [arxiv:2212.09802 eq. 6]. A segment consistency term asks the rays of one 2D segment to agree on a class 2212.09802 eq. 7. Gradients from the label losses are blocked before the density, so label noise cannot reshape the geometry [arxiv:2212.09802 §3.3].

• Author-reported on ScanNet, held-out views, 65.2 mIoU and 58.9 PQ-scene. The labeller alone gets 46.7 mIoU 2212.09802 tab. 1. HyperSim gives 60.1 PQ-scene 2212.09802 tab. 1.
• Test-time augmentation is load-bearing. Without it, ScanNet falls to 63.1 mIoU and 55.2 PQ-scene. Removing the gradient blocking gives 61.6 and 53.7 2212.09802 tab. 2.
• Cost: 450k iterations, about 10 hours on one A6000, and the pipeline needs offline pose estimation and label generation 2212.09802 §3.3 2212.09802 §4.1.3.
• The released code is CC BY-NC-SA 4.0, and the augmentation code is in a separate fork that is not in the repository README.md. Use the paper for the method and re-implement the augmentation step.
• The labeller limits the label set to the classes it was trained on 2212.09802 §4.1.3.

Card: Panoptic Lifting.

When to use which

Implementation sketch

The shared core of the two image models is the step that turns per-query masks into one panoptic map. Names are ours. The mask-wise rule of Panoptic SegFormer is the more involved one; the DETR rule is a pixel argmax.

# Panoptic map from N query masks, mask-wise rule  [arxiv:2109.03814 alg. 1]
# Input:  cls (N,) class ids, probs (N,), mask_logits (N, H, W), is_thing (N,)
# Output: class map (H, W), instance-id map (H, W)

def merge_masks(cls, probs, mask_logits, is_thing, t_cnf=0.25, t_keep=0.6):
    # Binarise each mask at 0.5  [arxiv:2109.03814 alg. 1]
    binary = mask_logits > 0.5
    # Quality: mean of the positive values; the authors leave the average
    # undefined, and the code adds 1 to the count  [arxiv:2109.03814 eq. 6]
    quality = (binary * mask_logits).sum((1, 2)) / (binary.sum((1, 2)) + 1)
    # Score: class probability times squared quality  [arxiv:2109.03814 eq. 6]
    score = probs * quality ** 2
    cls_map = zeros(H, W); inst_map = zeros(H, W); next_id = 1
    # Visit masks from most to least confident  [arxiv:2109.03814 alg. 1]
    for i in argsort(-score):
        if score[i] < t_cnf: continue
        # Keep only pixels no earlier mask has claimed  [arxiv:2109.03814 alg. 1]
        kept = binary[i] & (cls_map == 0)
        if kept.sum() < t_keep * binary[i].sum(): continue
        cls_map[kept] = cls[i]
        if is_thing[i]: inst_map[kept] = next_id; next_id += 1
    return cls_map, inst_map

# Pixel-wise alternative used by DETR  [arxiv:2005.12872 §4.4]
# Each pixel takes the mask with the highest score; no overlaps remain.
label_map = argmax(mask_scores, axis=query)     # (H, W)

Contradictions

• Panoptic FPN baseline. The Panoptic SegFormer table lists Panoptic FPN with ResNet-50 at 41.5 PQ after 36 epochs 2109.03814 tab. 1. The DETR table lists a retrained PanopticFPN++ at 42.4 PQ, with DETR's augmentation and an 18x schedule 2005.12872 tab. 5. The two rows use different training recipes, so the gap is not a result about the architecture.
• Merge rule for the same DETR model. The DETR paper reports 43.4 PQ with a pixel-wise argmax 2005.12872 §4.4 2005.12872 tab. 5. Panoptic SegFormer reports 44.7 for that model with its mask-wise merge 2109.03814 §4.3. A DETR number should name the merge rule.
• DETR epochs. The Panoptic SegFormer main table and ablation list DETR at 325 epochs, but its appendix lists 500 2109.03814 tab. 1 2109.03814 app. B.
• PQ is not one quantity. The COCO results are per-image PQ on single images 2109.03814 §4.2. Panoptic Lifting reports scene-level PQ, computed by tiling each scene's predictions and ground truth 2212.09802 §4. Panoptic Neural Fields report PQ on rendered views of KITTI 2205.04334 tab. 4. A 58.9 PQ-scene and a 49.6 COCO PQ are not comparable numbers.
• Stuff and things by query. Panoptic SegFormer says that boxes are a poor representation for stuff 2109.03814 §1. The DETR panoptic extension trains boxes for stuff too, because its matching depends on boxes 2005.12872 §4.4. The two designs answer the question differently.

Papers

• Panoptic SegFormer: separate thing and stuff queries, a deeply supervised mask decoder and mask-wise merging; reports COCO and ADE20K PQ.
• End-to-End Object Detection with Transformers: DETR, with a two-step mask head for its panoptic extension and a pixel-wise argmax at inference.
• Panoptic Neural Fields: per-object neural fields in tracked 3D boxes with a background model, trained on driving scenes.
• Panoptic Lifting: one neural field fitted to noisy 2D panoptic masks, with test-time augmentation and an injective ID assignment across views.