A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model

In plain words

A segmentation model labels every pixel of a picture, for example "road", "sky" or "dog". Teaching it new labels is expensive, because each new class needs pixel-level drawings. Image-text models such as CLIP can recognise many concepts from pictures and words alone, but they judge whole images, not single pixels.

This paper bridges the two. It first finds candidate regions that look like objects or surfaces, using a segmenter trained on the classes it has labels for. Then it cuts out each region, blanks out the rest, and asks CLIP which name fits best. The names are plain text, so new classes can be typed in when the system runs, with no new drawings. The authors' point is that this simple pipeline is a strong starting point, better than the usual one-network pixel classifier when both are built on CLIP.

TL;DR

Two stages: a mask proposer trained on seen classes, then CLIP classifies each masked crop against text prompts, and the per-mask scores are summed into a pixel map. The approach needs no per-class training for unseen names. Trained only on COCO-Stuff, it beats the FCN baseline in the cross-dataset setting. Author-reported zero-shot hIoU is 37.8 on COCO-Stuff and 77.5 on Pascal VOC. Use it as a baseline; the paper and code leave several settings unstated.

Problem

Segmentation datasets have tens to hundreds of classes, because pixel labels are costly §1. Zero-shot segmentation methods that use only segmentation labels generalise poorly §1. CLIP is trained at image level, while a fully-convolutional segmenter (FCN) classifies each pixel with a learned linear layer. The authors found that applying CLIP inside an FCN gives unsatisfactory results because of this granularity mismatch §1; §5; app. B.

The paper targets open-vocabulary segmentation: test classes are named in text, and training may use extra data beyond segmentation labels §3.1. Two settings are evaluated. In the zero-shot setting, training labels cover seen classes only and the test set contains seen and unseen classes §3.1. In the cross-dataset setting, the model is trained on COCO-Stuff and tested on other datasets without fine-tuning §3.1.

Method

The pipeline has two stages. Stage 1 proposes class-agnostic masks. Stage 2 classifies each masked region with CLIP and assembles the pixel map §4; fig. 1.

1. Mask proposals §4.1. Three generators were compared: GPB-UCM, Selective Search and MaskFormer. MaskFormer is the default, because the authors found its masks trained on seen classes transfer to unseen classes §4.1; tab. 5; tab. 6. The MaskFormer used has 100 queries §6.2.
2. Two strategies for region classification §4.2.1. (a) Apply the frozen CLIP image encoder to each masked crop. No training is needed, but seen-class training data is unused. (b) Retrain the image encoder inside MaskFormer, with the frozen CLIP text embeddings of the seen class names as fixed classifier weights. The encoder is pushed toward CLIP's text space, so unseen names can be scored later §4.2.1. The two strategies are combined by default §4.2.1; tab. 7.
3. Crop and region image §4.2.1; app. C.1; fig. 3; fig. 4. Each mask is binarised at 0.5, the bounding box is cropped, the background is filled with a constant, and the crop is resized to 224 by 224. The paper compares several background fills; the mean value gives the best result among those that work tab. 12.
4. Region probability eq. 1. For a region crop, the CLIP cosine similarity to each class text embedding is multiplied by a temperature of 100, then a softmax runs over the class names §4.2.1; eq. 1.
5. Class prompts §4.2.2. Two options are given. A hand-crafted option reuses CLIP's 80 ImageNet prompt templates, selected on training data §4.2.2. A learned option makes the prompt prefix a set of trainable vectors while the class name stays fixed §4.2.2; app. C.2.
6. Mask prediction assembly §4.3; eq. 2. Each pixel's score for a class is the sum over proposals of the proposal's mask value times its class score. The pixel gets the class with the highest score §4.3; eq. 2.
7. Training §4.2.1; §6.2. MaskFormer is trained on seen classes only. Its per-query classifier uses the CLIP text embeddings (with the optional learned prefix) as fixed weights, with a learned extra "no object" embedding §4.2.1; code: mask_former/modeling/clip_adapter/adapter.py#L178-L184.

# Two-stage open-vocabulary segmentation, inference  [§4] [fig. 1]
# Input:  image (3, H, W); class_names (list of C strings)
#         seen_flags (C,): 1 for training classes, 0 otherwise
# Output: class scores (C, H, W)

def segment(image, class_names, seen_flags):
    # 1. Propose Q = 100 masks and class logits per query  [§4.1] [§6.2]
    mask_logits, query_logits = proposal_generator(image)
    # (Q, H, W) and (Q, C + 1); the last column is "no object"
    query_probs = softmax(query_logits)[:, :-1]  # (Q, C)
    mask_probs = sigmoid(mask_logits)            # (Q, H, W)
    # 2. Binarise each mask at 0.5 and crop its bounding box  [§4.2.1]
    # 3. Fill the background with the CLIP mean colour, resize to 224
    #    [app. C.1]
    # 4. CLIP probability per region: cosine x 100, softmax over names
    #    [eq. 1]
    # Regions with no pixel above 0.5 get a neutral factor of 1
    #    [code: zero_shot_mask_former_model.py#L255-L257]
    region_probs = clip_region_probs(image, mask_probs, class_names)  # (Q, C)
    # 5. Combine the two probabilities per class with weight w
    #    [code: zero_shot_mask_former_model.py#L247-L272]
    # The paper gives no formula; w is 0.8 on COCO-Stuff in the config.
    w = 0.8
    seen = seen_flags[None, :]
    trained_part = query_probs ** w * region_probs ** (1 - w)
    unseen_part = query_probs ** (1 - w) * region_probs ** w
    class_probs = seen * trained_part + (1 - seen) * unseen_part
    # 6. Pixel score: sum over proposals of mask times class score
    #    [§4.3] [eq. 2]
    #    [code: zero_shot_mask_former_model.py#L277]
    return einsum("qc,qhw->chw", class_probs, mask_probs)

# Training of the query classifier  [§4.2.1] [§4.2.2]
# Input:  image batch; ground truth for seen classes only
# Output: scalar loss; gradients reach the mask head and the learned prefix

def training_loss(images, targets):
    # 1. Text embedding per seen class from frozen CLIP  [§4.2.1]
    # Prompt = learned prefix vectors + class-name tokens  [§4.2.2]
    #    [code: text_prompt.py#L88-L121]
    class_text = text_encoder(learned_prefix, seen_class_names)  # (C, 512)
    # 2. Add a learned "no object" embedding as one extra row
    #    [code: adapter.py#L178-L184]
    class_text = concat(class_text, no_object_embedding)  # (C + 1, 512)
    # 3. Map each query to the CLIP space with a 2-layer MLP
    #    [code: zero_shot_transformer_predictor.py#L49-L51]
    query_embed, mask_logits = mask_head(image)  # (Q, 512), (Q, H, W)
    # 4. Cosine similarity x 100 gives the query logits
    #    [code: zero_shot_mask_former_model.py#L171-L174]
    query_logits = 100 * normalise(query_embed) @ normalise(class_text).T
    # 5. Hungarian matching, then the mask-classification losses
    #    [code: mask_former_model.py#L89-L100]
    return mask_former_losses(query_logits, mask_logits, targets)

Hyperparameters

Values in the paper are cited to the paper. Values from the released code are cited to the code. "not stated" means neither source gives the value.

Results

Author-reported. Trained on COCO-Stuff, or on the stated seen classes; CLIP ViT-B/16 unless stated §6.1–§6.4. Table 4 varies the backbone and pre-training data, so its rows are not directly comparable to Tables 2 and 3.

Cross-dataset setting, mIoU, trained on COCO-Stuff, no fine-tuning tab. 1:

COCO-Stuff zero-shot setting, 156 seen and 15 unseen classes tab. 2:

Pascal VOC 2012 zero-shot setting, 15 seen and 5 unseen classes tab. 3:

Backbone and pre-training data on Pascal VOC, zero-shot tab. 4. The ImageNet rows use ImageNet-1K class names encoded by a text-only encoder (SimCSE, footnote in §6.4):

Key ablations, COCO-Stuff zero-shot setting:

• Mask proposal source, direct CLIP classification of masked crops, hIoU / mIoU unseen tab. 5: GPB-UCM 10.9 / 11.6; Selective Search 11.0 / 13.3; MaskFormer 28.2 / 29.7.
• Proposal generalisation, mIoU, oracle labelling of proposals tab. 6: COCO-Stuff to COCO-Stuff 69.4; ADE20K to COCO-Stuff 62.5; ADE20K to ADE20K 71.6; COCO-Stuff to ADE20K 64.4.
• Region classifier tab. 7: retrained vision encoder 8.7 hIoU (seen 38.7, unseen 4.9); CLIP vision encoder 28.2 (26.8, 29.7); ensemble 37.8 (39.3, 36.3).
• CLIP backbone, manual prompt, hIoU / mIoU unseen tab. 8: ResNet-50 15.2 / 16.0; ResNet-101 13.8 / 12.5; ViT-B/32 15.3 / 15.7; ViT-B/16 18.3 / 19.5.
• Background fill, hIoU tab. 12: keep context 9.3; zero 17.2; mean 18.3; learnable pixel prompt and mask token: failed to train stably.
• Learned vs hand-crafted prompt, hIoU tab. 14: manual 18.3; learned 28.2.
• Prompt length and samples per class, unseen accuracy tab. 13: best is length 32 with 32 samples per class (32.8).
• Supervised upper bound, COCO-Stuff tab. 9: MaskFormer trained on all 171 classes reaches hIoU 49.4 and unseen mIoU 62.6. This paper reaches 37.8 hIoU and 36.3 unseen mIoU, and 41.5 and 43.6 with self-training. Gap between thing and stuff unseen classes is 14.9 here, versus 9.5 for the supervised model tab. 10.
• Seen/unseen splits tab. 16: the official split gives hIoU 37.8; the three random splits give 30.9 to 36.6.

Paper vs code

Official code: the repository linked in the abstract, github.com/MendelXu/zsseg.baseline, at commit d77c3af72cc706125698d2538f3b14ba3d9b58aa. The owner handle MendelXu appears to be the first author's account (Mengde Xu), and the MIT LICENSE names the same owner. The code builds on MaskFormer and Detectron2; Detectron2 is not in the repository. Citations below are to the commit above.

• Self-training is not described in the paper. The paper reports "+ST" numbers but never says how the pseudo-labels are made §6.4; tab. 2–3. The code does it in an evaluator: on training images, pixels whose ground truth is a seen class take that ground truth, the rest take the model's prediction, optionally with seen-class probabilities zeroed, and low-confidence pixels are rejected pseudo_sem_seg_evaluation.py. The defaults are with_prior True and reject threshold 0.0 config.py. No released config turns this on, and tools/self_training.sh is empty.
• Ensemble rule. The paper says the two strategies are "ensembled" but gives no formula or weight §4.2.1. The code uses a per-class geometric mix with weight w. Seen classes use the trained classifier with weight w and CLIP with 1 − w. Unseen classes swap these zero_shot_mask_former_model.py. The released weights are 0.8 for COCO-Stuff and 0.7 for VOC.
• Mask assembly normalisation. Equation 2 divides by the sum of the mask values eq. 2. The code has no such division zero_shot_mask_former_model.py. The division is the same for every class at a pixel, so the argmax is unchanged, but the scores differ in scale.
• Mask-classification batch. The paper says the batch is 32 per GPU on 8 GPUs §6.2. The base config sets IMS_PER_BATCH 32 Base-COCOStuff164K-171.yaml. Detectron2 usually treats this field as the total batch, but Detectron2 is not in the repository, so the total batch is not verified here.
• Cross-dataset training. The paper gives 120K iterations §6.2. The repository contains only test-only configs for cross-dataset evaluation cross_dataset_test_only.yaml, so the training run is not reproducible from it.
• Prompt choice. The paper says the best prompt is chosen from the 80 ImageNet templates on training data §4.2.2. The code averages all templates instead text_prompt.py, and no selection script is in the repository.
• Prompt setting used in the headline runs. The paper's best learned prompt is 32 prefix vectors with 32 samples per class tab. 13. The main COCO-Stuff config uses 16 prefix vectors zero_shot_maskformer_R50_bs32_60k.yaml. The VOC config uses a predefined template, "a sculpture of a {}.", for the trained query classifier and the learned prompt only for the region classifier, with separate adapters zero_shot_maskformer_R50_bs16_20k.yaml. The paper does not state which prompt its VOC headline numbers use.
• Background fill colour. The paper's figure caption says "mean values of the dataset" app. C.1; fig. 4. The code uses CLIP's own pixel mean, scaled to 0–255 utils.py.
• Non-object class. The code adds a learned "no object" embedding to the class list during training adapter.py. It is not mentioned in the paper. At inference it is dropped before the CLIP softmax zero_shot_mask_former_model.py.
• Learned prompt schedule. The released prompt-tuning config is named ..._bs32_10k but sets MAX_ITER to 16000 zero_shot_proposal_classification_learn_prompt_bs32_10k.yaml. The paper gives 100 epochs, and an epoch count cannot be turned into iterations from the repository.
• Gradient clipping. The code clips the full model's gradients at 0.01 Base-COCOStuff164K-171.yaml. The paper does not mention it.
• Loss weights. The paper says the other settings follow MaskFormer §6.2. The code's weights are cited under Hyperparameters.
• Expansion ratio. The paper describes expanding each crop by a ratio r app. C.1 but never gives r. The code uses 1.0 config.py.
• FCN baseline. The FCN code (mask_former/ablation/zero_shot_per_pixel_model.py) and the sliding-window test were not audited against §5 and app. B.

No other discrepancies were found in the code for the two-stage method.

Pitfalls

• Reported numbers disagree inside the paper. Tables 2 and 9 disagree on the seen mIoU for "This paper + self-training" (39.6 vs 39.9); 39.6 is the value that reproduces 41.5 hIoU. The FCN+SW table in app. B gives 37.7 hIoU for "Ours", but Table 2 gives 37.8.
• Text and tables disagree in several places. Intro gives +19.6 hIoU over the previous best on COCO-Stuff §1; §6.4 gives +19.5 §6.4; Table 2 gives 19.6. §6.4 gives +37.7 hIoU and +46.8 unseen mIoU on VOC; Table 3 gives 37.8 and 46.9. Intro gives +19.6 on Pascal Context; Table 1 gives 19.5. App. C.2 gives +9.6 seen mIoU from learned prompts; Table 14 gives 9.5. Use the tables.
• Ensemble weight is a hidden hyperparameter. It changes results more than most listed settings and the paper never states it.
• Invalid masks. A proposal with no pixel above 0.5 gets a CLIP factor of 1, so only its trained-classifier term survives zero_shot_mask_former_model.py.
• Unseen-class mIoU is volatile. Appendix E shows the official split gives 37.8 hIoU, and the three random splits give 30.9 to 36.6 tab. 16. The claim that a more balanced unseen thing/stuff ratio gives higher hIoU holds only loosely: split 1 (ratio 0.36) scores 31.9, above split 2 (ratio 0.50, 30.9), even though it is farther from a ratio of 1.
• Pixel prompts and mask tokens fail. Learnable background fills did not train stably tab. 12. Do not build on them.
• Thing classes lag. Unseen thing classes score much lower than unseen stuff classes, a gap that self-training shrinks tab. 10. Expect poor results on object categories.
• Numerical sensitivity. Mask-to-crop uses a hard 0.5 threshold, so small objects can vanish. Each proposal costs one CLIP forward pass at 224 by 224, which grows with the number of proposals (100 per image) §6.2; code: adapter.py#L156-L175.
• Detectron2 and old NumPy. The repository needs Detectron2. The pseudo-label evaluator uses np.int and np.float pseudo_sem_seg_evaluation.py, which recent NumPy releases no longer provide.

Lineage

• Builds on: MaskFormer, arXiv:2107.06278 for the mask proposal and classification head; the paper replaces the learned class weights with CLIP text embeddings §4.1; §4.2.1.
• Builds on: CLIP, arXiv:2103.00020 for the image and text encoders §3.2.
• Uses prompt learning from Liu et al. (2021) and Zhou et al. (2021), bib keys liu2021pre and zhou2021learning in the paper §4.2.2.
• Compared against: SPNet (Xian et al. 2019), ZS3 (Bucher et al. 2019), CaGNet (Gu et al. 2020), STRICT (Pastore et al. 2021, arXiv:2104.11692) tab. 2–3. Concurrent work: LSeg (Li et al. 2022) and OpenSeg (Ghiasi et al. 2021, arXiv:2112.12143) §2.
• Superseded by: not stated in this wiki.