Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

In plain words

Most segmentation models only know the labels they were trained on. A model like CLIP understands any words you type, but it judges whole images, not regions. This paper asks how to get both: cut an image into regions, then name each region with any text.

The method splits the work in two. One network proposes candidate regions (masks). A second network, CLIP, names each region by cropping it out, blanking everything else, and comparing it with the words. The paper finds that CLIP is the weak link, because blanked-out crops look nothing like the photos it was trained on. So the authors teach CLIP to handle such crops. They do this with noun labels that come from the captions of ordinary photos, not from costly pixel labels, since captions name far more things than a fixed label list does. A further trick treats the blank areas as slots for learned tokens, which can be trained without changing CLIP itself.

The result is a segmenter for open vocabularies that does well on several benchmarks it was never trained on.

TL;DR

• Two-stage open-vocabulary segmenters are limited by CLIP, which classifies masked crops poorly §1.
• The paper fine-tunes CLIP on masked-region and noun pairs mined from COCO captions, which keeps CLIP's generalisation better than fine-tuning on fixed COCO-Stuff labels §3.2 §4.3.1.
• Mask prompt tuning replaces fully masked patch tokens with learnable prompts; it can run with CLIP frozen or on top of full fine-tuning §3.3 §4.3.2.
• Author-reported: 29.6 mIoU on ADE20K-150 (Swin-B, CLIP ViT-L/14, trained on COCO), 8.5 above the previous best zero-shot result tab. 1 §4.2.

Problem

Two-stage methods first produce class-agnostic masks, then classify each masked region with a pre-trained CLIP. The paper finds that the classifier is the bottleneck. With oracle (ground-truth) masks and an ordinary CLIP, mIoU on ADE20K-150 is 20.1%; with ordinary MaskFormer proposals and an oracle classifier it is 66.5% §1. CLIP is trained on natural images with little augmentation, while masked crops contain large blank areas and background noise §1 §3.1.

Fine-tuning CLIP on accurate COCO-Stuff masks with 171 fixed classes fixes the crops but hurts open-vocabulary transfer: 965K pairs drop A-847 mIoU by 2.0 points tab. 2, case (1) §4.3.1. The paper assumes an image-level caption vocabulary is more diverse and keeps CLIP's generalisation §1.

Assumptions stated in the paper: the segmenter can make class-agnostic proposals, and CLIP can transfer to masked proposals §1. The paper also notes that its "class-agnostic" MaskFormer is not fully class-agnostic, since the training classes define what counts as an object §3.1.

Method

Two-stage model §3.1

1. MaskFormer predicts N mask proposals. Each proposal gets a C-dimensional embedding, where C is the CLIP embedding width (512 for ViT-B/16, 768 for ViT-L/14). It is trained on COCO-Stuff with 171 classes, using CLIP text embeddings as classifier weights. A learnable "no object" embedding is added §3.1.
2. Mask-level class probabilities come from a softmax over the scaled cosine similarity between each mask embedding and each class text embedding. The paper writes the scale as τ but does not give its value §3.1.
3. A parallel CLIP branch crops each proposal to its tight bounding box, masks out background pixels, resizes to CLIP's input size, and classifies the crop §3.1.
4. The two branches are combined by a weighted geometric mean with weight λ in [0, 1]. The proposal-level scores are then fused into a semantic map with MaskFormer's own fusion step §3.1 app. A.3.

Mining mask-category pairs from captions §3.2

1. Run the class-agnostic MaskFormer on each image to get about 100 proposals §3.2 §4.3.1.
2. Extract the nouns of each caption with an off-the-shelf parser (NLTK is cited) and treat them as candidate classes §3.2.
3. Use CLIP to pair each noun with the proposal it matches best §3.2.
4. Using one caption per image gives 440K pairs with 12K unique nouns. Using all five captions gives 1.3M pairs with 27K nouns §3.2 tab. 2.

Fine-tuning CLIP §4.1

Only the CLIP image encoder is fine-tuned on the mined pairs; the text encoder is frozen. Three adaptation modes are compared:

• MPT only: start from OpenAI CLIP weights and train only the mask prompts.
• FT only: fine-tune all image-encoder weights.
• FT then MPT (the default and best result): fine-tune first, then train mask prompts on the fine-tuned model §4.1 app. A.5.

The paper does not state the training objective for the CLIP fine-tune §4.1.

Mask prompt tuning §3.3

1. A masked image is tokenised into N_p patch tokens of width E. A patch counts as masked only if every pixel in it is background. Partly covered patches keep their tokens, because boundary pixels matter for classification §3.3.
2. Masked tokens are replaced by learnable prompt tokens P, using the binary patch mask M_p §3.3 eq. in §3.3.
3. Prompts are added at deeper layers too, following deep visual prompt tuning. The default prompt depth is 3 §3.3 §4.1 app. A.5 tab. 6.

T \otimes M_p + P \otimes (1 - M_p) §3.3

The paper writes this formula inline in the text, and it is the only formula for the prompt step. It is not numbered in the source §3.3.

# Mask prompt tuning forward for one masked region  [§3.3]
# Input:  region (3, 224, 224); region_mask (224, 224) in {0, 1}
# Output: image embedding (E_out,)
def encode_masked_region(region, region_mask, prompts):
    # prompts: (prompt_depth, N_p, E), learnable; prompts[0] is the input
    # Tokenise the region into N_p = (224 / patch_size) ** 2 patches  [§3.1]
    patches = patch_embed(region)                       # (N_p, E)
    # Keep a patch if any of its pixels is foreground  [§3.3]
    keep = ceil(avg_pool(region_mask, patch_size))      # (N_p, 1) in {0, 1}
    # Replace fully masked patch tokens with the input prompt  [§3.3]
    patches = patches * keep + prompts[0] * (1 - keep)
    tokens = prepend_cls_token(patches)                 # (1 + N_p, E)
    for depth, block in enumerate(transformer_blocks, start=1):
        tokens = block(tokens)
        # Deep prompts: reapply after blocks 1 .. prompt_depth - 1  [app. A.5]
        if depth < prompt_depth:
            tokens[1:] = tokens[1:] * keep + prompts[depth] * (1 - keep)
    return output_projection(layer_norm(tokens[0]))

Hyperparameters

Values in the paper, with the shipped code noted where it differs. "Not stated" means neither the paper nor the code sets it in a place we checked.

Segmenter (Swin-B, CLIP ViT-L/14, the best model)

Segmenter (R101c, CLIP ViT-B/16, the small model)

CLIP adaptation (ViT-L/14)

Results

Author-reported. Setup: COCO-trained (COCO-Stuff with captions), evaluated zero-shot on ADE20K (A-150, A-847), Pascal Context (PC-59, PC-459) and PASCAL VOC (PAS-20) validation sets §4.1. Supervised specialist rows are trained on each test set and are shown for reference only tab. 1 §4.2. The paper says OVSeg is not directly comparable with them §4.2.

Main comparison (mIoU, %) tab. 1

Source: tab. 1. The SPNet and ZS3Net PAS-20 numbers are taken from the LSeg paper; the ZegFormer PAS-20 number was recalculated by the authors tab. 1 caption.

Ablation on mask-category pairs (mIoU, %) tab. 2. Baseline: MaskFormer Swin-B with original CLIP ViT-L/14.

Source: tab. 2 §4.3.1.

Ablation on mask prompt tuning (mIoU, %) tab. 3. Baseline: MaskFormer Swin-B with CLIP ViT-L/14.

Source: tab. 3 §4.3.2. Case (c) is the default FT-then-MPT model app. A.5.

Supporting appendix results (author-reported)

• Ensemble on A-150, Swin-B + ViT-L/14. Baseline: MaskFormer only 19.6, CLIP only 14.3, ensemble 21.8. OVSeg: 19.6, 25.1, 29.6 app. A.3 tab. 4.
• Combining MPT and FT tab. 5 app. A.5: FT then MPT, the default, gives 9.0 A-847 and 29.6 A-150. MPT then FT gives 8.5 and 28.1. Simultaneous MPT and FT gives 8.8 and 29.0.
• Prompt depth on A-150 tab. 6 app. A.5: without FT, depths 1, 3, 6, 12 give 25.7, 26.5, 26.8, 26.8. With FT, they give 29.3, 29.6, 29.4, 29.3.
• MPT vs. VPT on A-150 tab. 7 app. A.6: baseline 21.8, MPT 26.5, VPT 25.5. VPT used 50 extra tokens, depth 6, and about 40% more compute.
• Training-pair source, A-150 tab. 8 app. A.7: GT COCO-Stuff 23.0, captions only 28.8, both 26.7.
• Class IoU on A-150: average 37.6% on seen classes, 21.9% on unseen classes app. A.8.
• Inference cost (Swin-B + ViT-L): MaskFormer about 0.2 s per image and CLIP about 0.6 s per image on an NVIDIA A5000 app. A.9.
• Crop choice: a crop that keeps background pixels gives 13.8% mIoU, 6.3 points below masked crops app. A.1 fig. 7.

Qualitative results. The model segments user-defined queries such as "Saturn V", "Oculus" and "golden gate" §4.4.1 fig. 5. The paper also shows that the A-150 protocol can mark a reasonable synonym as wrong (for example "skyscrapers" for "building") §4.4.2 fig. 6.

Paper vs code

Code. Official repository facebookresearch/ov-seg at commit 36f49d496714998058d115ffb6172d9d84c59065 [code_commit]. The paper's project page links it [ovseg.tex#L87], and its owner is Meta, the authors' affiliation [ovseg.tex#L75-L86]. The repository's LICENSE is CC BY-NC 4.0, which is non-commercial. We use the code only to check the paper's numbers. We do not quote it, and nothing in this card is copied from it. Implement from the paper and this card, not from the code.

What we checked: the three shipped configs, the CLIP adapter, the ensemble, the mask-prompt forward pass, the CLIP fine-tune and mask-prompt scripts, the masked training loop, and the text templates. We did not review the MaskFormer head and transformer predictor line by line.

Differences:

• MPT learning rate. Paper: 2e-2 §4.1. Shipped script: --lr 0.05 coco_proposal_1cap_mask_prompt_tuning_VitL.sh. The paper's CLIP fine-tune learning rate, 5e-6, matches the script coco_proposal_1cap_finetune_VitL.sh.
• MPT batch size. Paper: 256 §4.1. Shipped script: 32 per GPU on 4 processes, i.e. 128 in total coco_proposal_1cap_mask_prompt_tuning_VitL.sh. The paper names no GPU count, so the 256 may refer to different hardware.
• Fine-tune batch for ViT-B/16. Paper: "a larger batch size 1024" than the ViT-L setting app. A.4. Shipped script: 128 per GPU on 4 processes, i.e. 512 in total coco_proposal_1cap_finetune_VitB.sh. The paper also says "ViT-L/16" where it means ViT-L/14 app. A.4.
• Ensemble weight. Paper: λ = 0.6 for PAS-20, PC-59 and PC-459, 0.7 for A-150 and A-847 app. A.3. Both shipped configs set 0.7 ovseg_swinB_vitL_bs32_120k.yaml ovseg_R101c_vitB_bs32_120k.yaml. The search script sweeps 0.6 to 0.8 search_thr_ensemble_w.sh. The per-dataset 0.6 is not in any shipped config.
• Mask threshold. The paper gives no threshold for turning proposal probabilities into crops. The code uses 0.4 for Swin-B ovseg_swinB_vitL_bs32_120k.yaml, 0.5 for R101c ovseg_R101c_vitB_bs32_120k.yaml and 0.35 in the Swin-B demo config ovseg_swinB_vitL_demo.yaml.
• Background fill. The paper says background pixels are "set to zeros" §1 §3.3. The code fills with the CLIP mean colour adapter.py, which becomes zero after normalisation adapter.py. The two agree in effect, but differ in what "zero" means.
• Prompt depth. Paper: "depth 3" app. A.5. In the code, depth 3 means three prompt sets: one at the input and one each after blocks 1 and 2 model.py model.py.
• Prompt inference. The configs enable mask prompts at test time (MASK_PROMPT_FWD: True) ovseg_swinB_vitL_bs32_120k.yaml. The README evaluates the FT-then-MPT checkpoint with MASK_PROMPT_FWD False in the ablation section GETTING_STARTED.md. The paper does not say whether prompts are used at test time.
• R101c initialisation. Paper: ImageNet-1K app. A.4. The config loads detectron2://DeepLab/R-103.pkl ovseg_R101c_vitB_bs32_120k.yaml. The paper does not name a weight file, so we cannot confirm it is the same pre-training.
• Pair mining. The paper describes the mining step in §3.2. The repository has no mining code. The shipped CSVs and their image archives are referenced in the README README.md, not included.
• Loss. The paper does not state the CLIP fine-tune objective. The code uses open_clip's ClipLoss in both the plain and masked training loops train.py train.py.
• Text templates. Matches: the 14 ViLD templates are the same list text_template.py. The code averages normalised text embeddings and re-normalises them text_template.py.
• Non-object embedding. The paper's learnable "no object" embedding §3.1 is concatenated to the class text embeddings adapter.py.
• Patch rule. Matches. A patch is masked only when all its pixels are background §3.3. The code average-pools the binary mask and takes the ceiling, so a patch is kept when any pixel in it is foreground model.py model.py.
• Ensemble formula. Matches: the code computes mask_cls^(1-λ) * clip_cls^λ ovseg_model.py, as in the paper §3.1.
• Learning schedule. Matches: cosine annealing with a linear warm-up scheduler.py. The paper gives no warm-up length; the code uses 100 steps.

Internal inconsistency in the paper. §4.2 says the Swin-B model reaches 55.5 mIoU on PC-59, while tab. 1 gives 55.7. The 55.7 figure matches the stated +13.6 gap over OpenSeg's 42.1 §4.2 tab. 1. Use 55.7 from the table.

Pitfalls

• Unstated temperature. The paper writes the class-score temperature as τ but never gives it §3.1. The code uses a fixed logit scale of 100, which is τ = 0.01 adapter.py. Use the code value or tune it.
• Unstated mask threshold. A threshold decides which proposal pixels become a crop. The shipped configs use 0.35 to 0.5 (see Paper vs code). Changing it changes results, and the repo's search script sweeps it.
• Unstated pairing score. The paper says CLIP pairs "the most matching" proposal to each noun §3.2, but does not give the score. The noun parser is only "an off-the-shelf language parser" §3.2. Expect the exact mining to differ.
• Unstated objective. The fine-tune loss is not stated. The code uses a contrastive ClipLoss train.py. Reproducing the paper's CLIP numbers requires the same objective.
• Learning rate and batch inconsistencies. The MPT learning rate (2e-2 vs 0.05) and batch size (256 vs 128 on 4 GPUs) differ between the paper and the shipped script. Try 2e-2 and batch 256 first, and say which you used.
• Prompt depth is easy to misread. "Depth 3" means three prompt sets: one at the input and two in deeper blocks. A depth-2 implementation (input plus one block) would not match the paper's default model.py.
• Prompt initialisation. The paper says "randomly initialized" §4.1. The code uses normal initialisation with std 0.02 model.py. The paper's MPT-only results start from OpenAI weights §4.1; the shipped MPT script starts from a fine-tuned checkpoint instead coco_proposal_1cap_mask_prompt_tuning_VitL.sh. No MPT-only script is shipped.
• Fine-tuning overfits to fixed labels. Training on COCO-Stuff's 171 GT classes drops A-847 by 2.0 points tab. 2. Captions help only with proposals, not with GT masks tab. 2 case (2) vs (3).
• Five captions are not better. Using five captions per image gives 0.2 points less on A-150 and PC-59 than one caption, at about five times the training cost §4.3.1.
• Compute is dominated by CLIP. CLIP runs once per proposal, about 100 per image, so inference is about 0.6 s on an A5000 for ViT-L/14 app. A.9.
• Unseen classes score lower. Seen-class IoU averages 37.6%, unseen 21.9% on A-150 app. A.8. Headline numbers over-state performance on unfamiliar vocabularies.
• Protocol penalises synonyms. The paper shows A-150 scoring a reasonable synonym as wrong §4.4.2 fig. 6. Expect some errors that a human would not count.
• Not stated. GPU count, warm-up length for the CLIP fine-tune, Adam betas and epsilon, the exact augmentation for the CLIP fine-tune, and the test-time resolution for CLIP crops. The code uses params.py defaults for betas and epsilon, and these depend on the model name params.py.
• Code is non-commercial. Use the paper, not the repo, when building a commercial product (see Paper vs code).

Lineage

• Builds on: two-stage open-vocabulary segmentation (ZSSeg, arXiv:2112.14757; ZegFormer) and MaskFormer, with a CLIP text classifier §3.1. The modified MaskFormer follows ZSSeg §3.1.
• Builds on: CLIP, for masked-region classification §3.1. Also visual prompt tuning (arXiv:2203.12119) for the mask prompts §3.3.
• Related: RegionCLIP adapts CLIP on region crops for detection; the paper differs by masking the image and learning the blank areas §2.
• Compared against: OpenSeg, the main proposal-based baseline tab. 1. Card: OpenSeg. Also SPNet, ZS3Net, LSeg, LSeg+, SimBaseline and ZegFormer tab. 1.
• Superseded by: not stated in this wiki.