Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

In plain words

Panoptic segmentation labels every pixel of a photo. Countable things such as people or cars each get their own identity, and amorphous stuff such as sky or grass gets one label per kind. Earlier transformer models handled both kinds of content with one shared set of learned slots, which trained slowly and gave blurry object edges.

Panoptic SegFormer gives things and stuff separate sets of slots, so the two kinds do not get in each other's way. The masks come from the attention maps that the transformer already computes, and the intermediate layers are supervised too, so those maps focus on the right regions early. When the final picture is assembled, the most confident masks claim their pixels first, instead of each pixel taking whichever label scores highest. The authors report that this trains in fewer epochs than DETR-style baselines and gets higher accuracy on the COCO benchmark.

TL;DR

Panoptic SegFormer is a transformer panoptic segmenter. Things and stuff use separate query sets, a deeply supervised mask decoder turns attention maps into masks, and mask-wise merging replaces pixel-wise argmax. Author-reported: 49.6 PQ on COCO val with ResNet-50 after 24 epochs (MaskFormer: 46.5 PQ after 300), and 56.2 PQ on COCO test-dev with Swin-L. The official code is Apache-2.0 and differs from the paper in several places (see Paper vs code).

Problem

DETR-style panoptic models have three named weaknesses. Training takes a long time to converge §1. Self-attention cost grows with the square of the sequence length, which limits the feature resolution, and the FPN-style mask head then gives low-fidelity boundaries §1. Stuff, which is amorphous, is represented by bounding boxes, which the authors call suboptimal for it §1 §3.3.1. Pixel-wise argmax at inference also produces false-positive regions when a mask has extreme values §3.5 app. B.

The setting is COCO 2017 without external data: 118K training images and 5K validation images, with 80 thing classes and 53 stuff classes §4.1. ADE20K (100 things, 50 stuff) is a second benchmark §4.1.

Method

Overview. A backbone produces three feature maps at 1/8, 1/16 and 1/32 of the input resolution. Each is projected to 256 channels and flattened, and a deformable transformer encoder refines the combined tokens §3.1 §3.2 fig. 2. The model has N_th = 300 learned thing queries and N_st stuff queries, one per stuff class §3.1 app. A. A location decoder refines only the thing queries; an auxiliary box and class head supervises it during training and is discarded at inference §3.3.2. A mask decoder then processes thing and stuff queries alike and outputs a mask and a class per query §3.3.3.

Query decoupling. Thing queries are matched to ground-truth objects with bipartite (Hungarian) matching. Each stuff query is fixed to one stuff class, so no matching is needed for stuff §3.3.1 §3.4 fig. 3.

Mask decoder. At each mask-decoder layer the attention maps are split by feature level, the two coarser levels are bilinearly upsampled to the 1/8 resolution, and the three are concatenated along the head axis eq. 1 eq. 2. A small head with about 200 parameters turns the fused maps into one binary mask per query §3.3.3 app. B. Every decoder layer receives the mask loss (deep supervision) §3.3.3. Classes come from a fully connected layer on the refined query at each layer §3.3.3. The code computes the thing classes differently; see Paper vs code.

Loss. The total loss is a weighted sum of a thing term and a stuff term, with weights that sum to 1 eq. 3 app. A. The thing term adds a detection loss to focal classification and Dice segmentation losses summed over decoder layers eq. 4. The stuff term is focal classification plus Dice over decoder layers, with the fixed stuff assignment eq. 5. The matching cost for things is the sum of classification, detection and segmentation costs §3.4.

Inference: mask-wise merging. Instead of a pixel-wise argmax, each predicted mask gets a confidence score that multiplies its class probability by a power of its mask quality eq. 6 app. A. Masks are then placed in descending score order and keep only the pixels not yet claimed; a mask is dropped if its score is below a threshold or too little of it survives alg. 1 app. A.

Pseudocode (our rendering of the architecture, with the paper's equations):

# Panoptic SegFormer forward pass  [§3.1] [fig. 2]
# Input:  image (B, 3, H, W)
# Output: per-query class scores and masks, for things and stuff

def forward(image):
    # Backbone stages C3, C4, C5 at strides 8, 16, 32  [§3.1]
    feats = backbone(image)
    # Project each map to 256 channels and flatten into tokens  [§3.1]
    tokens = concat([flatten(project(f)) for f in feats])  # (B, L, 256)
    # Deformable encoder refines the multi-scale tokens  [§3.2]
    memory = encoder(tokens)                               # (B, L, 256)
    # Location decoder refines the 300 thing queries only  [§3.3.2]
    thing_q = location_decoder(thing_embed, memory)       # (B, 300, 256)
    # One learned stuff query per stuff class  [§3.3.1] [app. A]
    stuff_q = stuff_embed                                  # (B, N_st, 256)
    # Mask decoder turns attention maps into masks and classes  [§3.3.3]
    masks, classes = mask_decoder(thing_q, stuff_q, memory)
    return masks, classes

# Mask from one decoder layer's attention maps  [§3.3.3] [eq. 1] [eq. 2]
# attn: (N, heads, L), where L holds the tokens of all three levels

def mask_from_attention(attn, level_shapes):
    # Cut the token axis back into the three feature levels  [eq. 1]
    level1, level2, level3 = split(attn, level_shapes)
    # Upsample the two coarser levels to the level-1 size  [eq. 2]
    fused = concat([level1, up2(level2), up4(level3)], axis=heads)
    # Mix the heads into one mask per query  [§3.3.3]
    return mix_heads(fused)                       # (N, 1, H/8, W/8)

# Training loss for things and stuff  [§3.4] [eq. 3] [eq. 4] [eq. 5]
# Hungarian matching for things uses classification, detection and
# segmentation costs  [§3.4]

def training_loss(preds, targets):
    # Match thing queries one-to-one with ground-truth objects  [§3.4]
    matches = hungarian(cost(preds.thing, targets.thing))
    # Detection loss from the location decoder  [eq. 4] [app. A]
    loss_things = lambda_det * detection_loss(matches)
    # Focal and Dice losses on the matched masks of every layer  [eq. 4]
    for layer in thing_decoder_layers:
        loss_things += lambda_cls * focal(layer, matches)
        loss_things += lambda_seg * dice(layer, matches)
    # Stuff queries use a fixed query-to-class assignment  [§3.4] [eq. 5]
    loss_stuff = 0
    for layer in stuff_decoder_layers:
        loss_stuff += lambda_cls * focal(layer, stuff_targets)
        loss_stuff += lambda_seg * dice(layer, stuff_targets)
    # Weights follow the things and stuff share of the image  [app. A]
    ratio = n_things / (n_things + n_stuff)
    # Weighted sum of the two parts  [eq. 3]
    return ratio * loss_things + (1 - ratio) * loss_stuff

# Mask-wise merging  [alg. 1] [eq. 6] [app. A]
# Input:  classes (N,), probs (N,), mask_logits (N, H, W)
# Output: semantic map (H, W), instance-id map (H, W)

def mask_wise_merge(classes, probs, mask_logits):
    # Binarize every mask at 0.5  [alg. 1]
    binary = mask_logits > 0.5                         # (N, H, W)
    # Mask quality: mean of the positive values, with +1 in the
    # denominator as in the code  [eq. 6]
    # [code: easymd/models/panformer/panformer_head.py#L1140]
    quality = (binary * mask_logits).sum((1, 2)) / (binary.sum((1, 2)) + 1)
    # Confidence = class probability times quality squared  [eq. 6]
    # alpha = 1, beta = 2  [app. A]
    scores = probs * quality ** 2                      # (N,)
    order = argsort(-scores)
    semantic = zeros(H, W)
    instance = zeros(H, W)
    next_id = 1
    # Visit masks from most to least confident  [alg. 1]
    for i in order:
        # Skip masks below the confidence threshold  [app. A] (t_cnf = 0.25)
        if scores[i] < t_cnf:
            continue
        # Keep only pixels that no earlier mask has claimed  [alg. 1]
        kept = binary[i] & (semantic == 0)
        # Drop the mask if too little of it survives  [app. A] (t_keep = 0.6)
        if kept.sum() < t_keep * binary[i].sum():
            continue
        semantic[kept] = classes[i]
        if classes[i] is a thing:
            instance[kept] = next_id
            next_id += 1
    return semantic, instance

Hyperparameters

Results

Author-reported throughout. COCO 2017 val with single-scale input, unless the row says test-dev. "PS" means Panoptic SegFormer; † marks ImageNet-22K pretraining. Epochs and parameters (#P, millions) and FLOPs (#F, billions) are as reported tab. 1.

COCO val tab. 1

COCO test-dev tab. 2

ADE20K val tab. 3

Instance segmentation, COCO test-dev, AP^seg tab. 4

Instance segmentation is obtained by discarding the stuff queries §4.2.

Ablation, COCO val, R50 tab. 5

Post-processing, Mask PQ, R50 §4.3

Other ablations, COCO val, R50 (all §4.3 unless stated)

• Location decoder depth (0, 1, 2, 6 layers): PQ 47.0, 47.7, 48.1, 49.6. A box-free variant that predicts mask centres instead of boxes gets 49.2 §4.3.
• Query decoupling versus joint matching: PQ 49.6 against 48.5, and PQ^st 42.4 against 39.5 §4.3.
• Using the first layers of the mask decoder only: PQ 48.8, 49.5, 49.6 and 49.6 for layers 1, 2, 3 and last, with FPS 10.6, 9.8, 9.3 and 7.8 §4.3.
• Backbones (PQ / PQ^th / PQ^st) app. D: ResNet-50 49.6 / 54.4 / 42.4; ResNet-101 50.6 / 55.5 / 43.2; PVTv2-B0 49.5 / 55.3 / 40.6; PVTv2-B2 52.5 / 58.5 / 43.4; PVTv2-B5 55.4 / 61.2 / 46.6; Swin-L 55.8 / 61.7 / 46.9.
• Heads in the mask decoder: 1 head 49.2 PQ, 8 heads 49.6 PQ app. B.
• Confidence exponents (alpha, beta): (1,0) 48.7; (0,1) 44.4; (1,1) 49.3; (1,2) 49.6; (1,3) 49.7 app. B.
• Thresholds (t_cnf × t_keep grid, R50): 48.8 to 49.7 for t_cnf 0.20–0.30 and t_keep 0.5–0.9; t_cnf 0.35 and 0.40 fall to 47.4–49.2 app. B.
• Robustness, COCO-C (16 corruptions on a 2000-image subset of val2017): the Swin-L model averages 47.2 PQ, which the authors say is 5.5 PQ above MaskFormer (Swin-L) §4.4.
• Deformable DETR as a panoptic baseline (50 epochs, with the DETR panoptic head): single-scale 40.6 PQ, multi-scale 46.3 PQ app. B.

Paper vs code

Checked against the official repository at commit cdb9b68 (Apache-2.0). Every item below is a discrepancy or an undocumented choice.

• Thing classes come from the location decoder, not the mask decoder. The paper says the mask-decoder FC predicts the category of each thing query §3.3.3. In the code the mask-decoder thing-class loss is multiplied by zero panformer_head.py. At inference, thing labels and scores come from the location decoder's sigmoid scores panformer_head.py panformer_head.py. Stuff classes do come from the mask decoder panformer_head.py.
• Loss weights. The paper gives λ_cls = 2, λ_seg = 1, λ_det = 1 app. A. The config uses a Dice weight of 2 base.py. The stuff classification is multiplied by an extra 2 on top of the focal weight of 2 panformer_head.py base.py, which gives an effective weight of 4 (derived from code).
• Mask decoder depth and attention. The paper says 6 vanilla cross-attention layers app. A. The code uses 4 layers for things and 6 for stuff, each with one tail layer base.py mask_head.py mask_head.py. Only the stuff head has self-attention base.py mask_head.py. The appendix says self-attention was dropped app. B.
• Mask head. The paper describes a 1×1 convolution on the fused maps §3.3.3. The code mixes each level's heads with a linear layer and ReLU, then applies a linear layer and a final ReLU across the three levels mask_head.py mask_head.py. Its outputs are therefore non-negative.
• Stuff overlap rule. The paper uses one keep threshold, t_keep = 0.6 app. A. The code keeps things at 0.6 but requires stuff to keep 0.8 of its area (overlap 0.2) panformer_head.py panformer_head.py.
• Quality score denominator. The paper's "average" is not defined, and its formula does not say what the denominator is. The code divides the sum over positive pixels by the count plus 1 panformer_head.py panformer_head.py.
• Thing-stuff ratio. The appendix says the ratio is set per image app. A. The code computes it over the whole batch panformer_head.py.
• Query filtering. The appendix describes repeated matching rounds to select thing queries app. A. The code runs min(3, 300 // number of ground-truth objects) rounds tools.py tools.py. The detection targets and the ratio use only the first round tools.py panformer_head.py. Only the mask-decoder queries use the union of all rounds panformer_head.py.
• Feature levels. The paper describes three levels §3.1. The config sets num_outs=4 for the neck and num_levels=4 for deformable attention base.py base.py. The mask head splits attention into three levels mask_head.py. Whether the neck really outputs a fourth level is not settled by reading the code, because the neck is an external library; a run would settle it.
• Thing candidates at inference. The paper does not say how many thing queries reach the mask decoder at inference. The code keeps the top 100 (query, class) pairs over 300 queries × 80 classes panformer_head.py base.py.
• Undocumented optimizer details. Weight decay 1e-4, AdamW, gradient clipping at 0.1, a 0.1 learning-rate multiplier for sampling offsets and reference points, and a frozen first backbone stage are all in the config but not the paper base.py base.py.
• PVTv2-B5 score. The paper's table gives 55.4 PQ tab. 1; the README gives 55.6 [README.md#L20]. The PVTv2-B5 config uses a thing and stuff threshold of 0.3 panformer_pvtb5_24e_coco_panoptic.py.
• Dataset constants. Things are taken to be labels below 80 and stuff labels from 80 onward, with the comment that this is not true for other datasets panformer_head.py panformer_head.py. The ADE20K results come from the paper; the repository has no ADE20K config.

No discrepancies were found in the loss matching form itself (Hungarian matching with focal, L1, GIoU and Dice terms), the positional encoding, the number of location-decoder layers (6) or the query counts (300 thing, 53 stuff).

Pitfalls

• Index naming in the mask equations. Equation 1 writes the three levels as A3, A4, A5 with resolution H/2^(i+2), while equation 2 uses A1, A2, A3 with upsampling factors 2 and 4 eq. 1 eq. 2. The consistent reading is level 1 = 1/8, level 2 = 1/16, level 3 = 1/32, which is what the code does mask_head.py.
• Threshold naming. Section 3.5 calls the confidence threshold t_cls, while the algorithm and appendix call it t_cnf §3.5 alg. 1 app. A.
• "Logits" are not logits. The mask values used in merging and in the 0.5 threshold are ReLU outputs, so they are never negative mask_head.py. Dice is computed on these raw values without a sigmoid dice_loss.py. The masks are upsampled to the original image size before thresholding panformer_head.py.
• Thing masks in the loss. Predicted thing masks are upsampled by 2 before the Dice loss, and the ground truth is resized to match panformer_head.py.
• Train and test see different thing queries. Training passes only queries matched in the filtering rounds to the mask decoder panformer_head.py. Inference passes the top-100 scores over all queries and classes panformer_head.py. The paper does not describe this gap.
• Duplicate queries at inference. One query can appear several times, once per class, because the top-k runs over query × class. Each copy then gets the same mask and a different label panformer_head.py panformer_head.py. Merging removes the duplicates, but an implementer who keeps all copies will see them.
• Thresholds act on the merged score. Both thresholds apply to the product of a probability and a squared quality, so changing the score function, for instance the +1 in the quality denominator, shifts where the thresholds bite panformer_head.py. The authors report that the result is not very sensitive to the threshold grid app. B.
• Constant count in the label array. arange(80, 134) yields 54 labels for 53 stuff queries. The extra label is never indexed, so the output is unaffected panformer_head.py.
• Hard-coded COCO layout. The thing and stuff split at 80 and the stuff range are constants in the head, not configuration panformer_head.py. Training on another dataset, such as ADE20K, needs these changed along with the query count.
• Hidden costs. The paper reports under 12 GB of GPU memory for training and more than 20 GB for MaskFormer with batch size 1 app. B. These are author measurements and were not checked here. The authors also state that deformable attention is slow and that small objects and high-resolution inputs remain hard §5.
• Thing and stuff scores come from different heads. Things are scored by the location decoder and stuff by the mask decoder panformer_head.py. Both enter the same merge score, so their distributions need not match. The card has not measured how far apart they are.
• Paper-internal inconsistency. DETR is listed with 325 epochs in the main table tab. 1 and the ablation tab. 5, but 500 epochs in the discussion table app. B.
• Version. This card follows arXiv version 4, the CVPR 2022 template source that wikillm fetch retrieved. The published date is that of version 1.

Lineage

• Builds on: DETR, whose panoptic head and matching the authors reuse as their base §1 app. A; Deformable DETR, whose deformable attention, box refinement and training settings are used for the encoder and location decoder §3.2 app. A.
• Compared against: Panoptic FPN, Panoptic FCN, MaskFormer, K-Net, Max-DeepLab, SOLOv2, QueryInst, HTC and SOLQ tab. 1 tab. 4. Mask-wise merging is compared with pixel-wise argmax and the heuristic procedure of Kirillov et al. §4.3 app. B.
• Superseded by: not stated in this wiki.