--- kind: concept slug: open-vocabulary-segmentation title: Open-vocabulary segmentation summary: >- Name each pixel or 3D point with a class given at test time. Covered: OpenSeg (72.2 PASCAL-20 mIoU), GroupViT (52.3 VOC 2012), OVSeg (94.5 PASCAL-20), ZSSeg (37.8 COCO-Stuff zero-shot hIoU), SAN (27.5 ADE-150), Feature 3DGS (0.782 Replica), ConceptFusion (0.446 3D mIoU, UnCoCo) and SparseGrasp (55/70 grasps, 3 views). Numbers are author-reported, with different backbones and data. tags: [computer-vision, semantic-segmentation, open-vocabulary, weak-supervision, vision-language, clip, prompt-tuning, 3d-scene-understanding, gaussian-splatting, grasping] papers: ["2112.12143", "2112.14757", "2202.11094", "2210.04150", "2302.07241", "2302.12242", "2312.03203", "2312.00732", "2412.02140"] related: [semantic-segmentation, vision-language-pretraining, zero-shot-learning, contrastive-learning, visual-prompting, vision-transformer, segmentation-evaluation, gaussian-splatting, knowledge-distillation, promptable-segmentation, robotic-grasp-generation, simultaneous-localization-and-mapping, surfel-mapping] status: draft updated_at: 2026-11-01 --- ## Overview Open-vocabulary segmentation assigns each pixel a label from a list of class names given at test time. The list does not have to match the training labels [arxiv:2112.12143 §1]. The usual obstacle is data: per-pixel models such as LSeg need pixel-level class labels, which are costly and limited to the vocabulary they were trained on [arxiv:2112.12143 §1]. Image-level vision-language models such as CLIP and ALIGN learn from cheap image-caption pairs but give one vector per image, so they lose where things are [arxiv:2112.12143 §1]. In the 2D variants, Feature 3DGS and ConceptFusion, class names and visual features live in the same embedding space. A name is embedded with a text encoder and compared with visual features, and each pixel or segment takes the name that scores highest [arxiv:2112.12143 §3.4] [arxiv:2202.11094 §3.3] [arxiv:2210.04150 §3.1] [arxiv:2112.14757 §4.3] [arxiv:2302.12242 §3] [arxiv:2312.03203 eq. 5]. The variants differ in how the visual side is built and where its supervision comes from: - **Mask proposals, caption grounding** (OpenSeg): masks are learned from class-agnostic labels, and caption words are aligned to regions. - **Text-only grouping** (GroupViT): no masks at all; patches are merged into segments by a caption-matching signal. - **Two-stage mask classification** (OVSeg): a mask generator proposes regions and a CLIP model, adapted to masked crops, names each one. - **Two-stage with seen-class proposals** (ZSSeg): a mask generator trained on seen classes only proposes regions, and a frozen CLIP names each masked crop. No captions are used. - **Side network on frozen CLIP** (SAN): a light side ViT predicts masks and attention biases, and the biases steer CLIP's last blocks to classify each mask in a single forward pass. - **Feature distillation into 3D Gaussians** (Feature 3DGS): a 2D teacher's features are stored per Gaussian and queried by similarity. - **Zero-shot features fused into a SLAM map** (ConceptFusion): no training. Pixel features mix a whole-frame embedding with mask-region embeddings, and are fused into map points that any query type can score. - **Sparse-view distillation for grasping** (SparseGrasp): MaskCLIP region features, compressed to 16 dimensions, are distilled into Gaussians from three views. The paper does not state how a text query reaches them, so it is the exception to the shared invariant as written. - **Class-agnostic 3D groups** (Gaussian Grouping): groups come from SAM masks and carry no names. Text is handled by a separate detector, so this is the exception to the shared invariant. ## Variants ### Region proposals with caption grounding (OpenSeg) The image is turned into a fixed set of mask proposals, each with a feature vector, instead of one vector or a dense per-pixel map [arxiv:2112.12143 §3]. The paper uses N = 128 proposals [arxiv:2112.12143 app. E]. The model is trained in three parts: - Masks: class-agnostic COCO panoptic masks supervise the proposals with a Dice loss, where each labelled mask is matched to its best proposal [arxiv:2112.12143 §3.1] [arxiv:2112.12143 §4.1]. - Alignment: nouns and adjectives from captions are matched to regions with a grounding loss, using a frozen BERT-Large text encoder [arxiv:2112.12143 §3.2] [arxiv:2112.12143 §4.1]. - Caption-only data (optional): a teacher trained on masks labels Localized Narratives with pseudo masks, and a student trains on human and pseudo labels [arxiv:2112.12143 §3.3]. Reported results (author-reported, zero-shot, mIoU) [arxiv:2112.12143 tab. 2]: - EfficientNet-B7, COCO and Localized Narratives: 70.2 on PASCAL-20. - Same model with prompt engineering: 72.2 on PASCAL-20 and 28.6 on ADE20k-150. - The paper compares the 72.2 with a 52.3 LSeg figure taken from the LSeg paper, which uses a different backbone and setup [arxiv:2112.12143 §4.3]. Card: [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143). ### Text-only grouping, no masks (GroupViT) GroupViT needs no mask labels. Its image encoder is a ViT-S whose Transformer layers are split into grouping stages [arxiv:2202.11094 §3.1] [arxiv:2202.11094 §4.1]. Learnable group tokens are concatenated with the segment tokens. At the end of each stage, a grouping block assigns every segment to one group and merges the segments of each group into a larger segment [arxiv:2202.11094 §3.1]. The headline model uses 64 group tokens and then 8 [arxiv:2202.11094 §4.1]. - Hard assignment: the assignment is one-hot in the forward pass and uses a straight-through gradient. The paper reports that soft assignment costs a large amount of mIoU [arxiv:2202.11094 §3.1] [arxiv:2202.11094 tab. 1]. - Training: image-text contrastive losses only, on CC12M and filtered YFCC [arxiv:2202.11094 §3.2] [arxiv:2202.11094 §4.1]. A multi-label loss adds nouns drawn from each caption, placed in short templates, as extra positives (K = 3) [arxiv:2202.11094 §3.2] [arxiv:2202.11094 §4.1]. - Zero-shot inference: each segment takes the label with the highest normalised similarity to the prompted label embeddings [arxiv:2202.11094 §3.3]. Segments with a low top probability are background, using thresholds of 0.9 on VOC 2012 and 0.5 on PASCAL Context [arxiv:2202.11094 §4.1]. Reported results (author-reported, zero-shot, mIoU) [arxiv:2202.11094 tab. 5]: 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context. The paper describes the VOC result as approaching a fully supervised DeiT row that uses class labels [arxiv:2202.11094 §4.4] [arxiv:2202.11094 tab. 5]. Background is the weak point. On PASCAL Context, classes such as ground, road and wall are grouped correctly but named wrongly, and their IoU is about 1.5 [arxiv:2202.11094 app. C.3]. Card: [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094). ### Mask-adapted CLIP for two-stage proposals (OVSeg) OVSeg is a two-stage model. A MaskFormer-style mask generator, trained on COCO-Stuff with 171 classes and with CLIP text embeddings as its classifier, predicts proposals with a class-probability head [arxiv:2210.04150 §3.1]. A parallel CLIP branch crops each proposal, masks out its background and classifies the crop. The two scores are combined by a weighted geometric mean [arxiv:2210.04150 §3.1]. The CLIP branch is adapted in three steps: - Mining pairs: run the class-agnostic mask generator on each image, extract nouns from the captions, and pair each noun with the proposal that CLIP matches best. One caption per image gives 440K pairs and 12K nouns [arxiv:2210.04150 §3.2] [arxiv:2210.04150 tab. 2]. - Fine-tuning: only the CLIP image encoder is trained on these pairs; the text encoder stays frozen [arxiv:2210.04150 §4.1]. - Mask prompt tuning: fully masked patch tokens are replaced by learnable prompts, and prompts are also added at deeper layers. This can run with CLIP otherwise frozen [arxiv:2210.04150 §3.3] [arxiv:2210.04150 §4.3.2]. Reported results (author-reported, mIoU) [arxiv:2210.04150 tab. 1]: with Swin-B and CLIP ViT-L/14, trained on COCO-Stuff-171, 9.0 on A-847, 29.6 on A-150, 55.7 on PC-59 and 94.5 on PAS-20. The ablation baseline, MaskFormer Swin-B with the original CLIP, scores 7.3, 21.8 and 51.4 on A-847, A-150 and PC-59 [arxiv:2210.04150 tab. 2] [arxiv:2210.04150 tab. 3]. Card: [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](/papers/2210.04150). ### Seen-class proposals with frozen CLIP crops (ZSSeg) ZSSeg is two-stage, like OVSeg, but it uses no captions. A MaskFormer is trained on the seen classes only and proposes class-agnostic masks [arxiv:2112.14757 §4.1] [arxiv:2112.14757 §6.2]. Each mask is labelled by two classifiers, and the two are combined by default [arxiv:2112.14757 §4.2.1] [arxiv:2112.14757 tab. 7]: - Frozen crop classifier: the CLIP image encoder, unchanged, classifies a crop of the masked region. - Retrained query classifier: the MaskFormer image encoder is retrained, with the frozen CLIP text embeddings of the seen names as fixed class weights. This pushes the encoder toward CLIP's text space, so unseen names can be scored later [arxiv:2112.14757 §4.2.1]. Each mask is binarised at 0.5, its bounding box is cropped, the rest is filled with a constant colour, and the crop is resized to 224 x 224 [arxiv:2112.14757 §4.2.1] [arxiv:2112.14757 app. C.1]. Among the fills the paper tests, the mean colour gives the best hIoU, 18.3 against 17.2 for zero [arxiv:2112.14757 tab. 12]. Class prompts come either from CLIP's 80 ImageNet templates, with the best one picked on training data, or from a learned prefix of trainable vectors [arxiv:2112.14757 §4.2.2]. In the paper's ablation the learned prefix raises hIoU from 18.3 to 28.2 [arxiv:2112.14757 tab. 14]. The card's Paper vs code section lists the prompt settings in the released configs. Reported results (author-reported, hIoU, COCO-Stuff zero-shot split of 156 seen and 15 unseen classes) [arxiv:2112.14757 tab. 2]: 37.8, with 39.3 mIoU on seen and 36.3 on unseen classes, against 20.9 for FCN. On Pascal VOC 2012 with 15 seen and 5 unseen classes [arxiv:2112.14757 tab. 3], it reports 77.5 against 39.7 for CaGNet, the best prior method without self-training. A MaskFormer trained on all 171 classes reaches 49.4 hIoU, which is the paper's supervised upper bound [arxiv:2112.14757 tab. 9]. Trained on COCO-Stuff with no fine-tuning, it reports 34.5, 47.7, 20.5 and 7.0 mIoU on Cityscapes, Pascal Context, ADE20K-150 and ADE20K-847, against 21.4, 28.2, 14.9 and 4.1 for FCN [arxiv:2112.14757 tab. 1]. Both branches are needed. The retrained encoder alone scores 8.7 hIoU and the frozen CLIP crops alone score 28.2, against 37.8 for the ensemble [arxiv:2112.14757 tab. 7]. Unseen thing classes lag unseen stuff classes: the gap is 14.9 here, against 9.5 for the supervised model [arxiv:2112.14757 tab. 10]. The ensemble weight is not given in the paper [arxiv:2112.14757 §4.2.1], and the crop expansion ratio is described but not set [arxiv:2112.14757 app. C.1]. Card: [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757). ### Frozen CLIP with a side network (SAN) SAN keeps CLIP frozen and adds a light side ViT with 8 blocks, width 240 and 6 heads. It has 8.4M trainable parameters in total [arxiv:2302.12242 §4.2] [arxiv:2302.12242 tab. 3]. The side network reads CLIP's earlier features, fused at fixed depths. The default split, nine fused CLIP blocks and three recognition blocks for ViT-B/16, scores best among the splits tested [arxiv:2302.12242 tab. 5]. The side network has two outputs [arxiv:2302.12242 §3]: - Mask proposals, from the inner product of pixel and query features [arxiv:2302.12242 eq. 1]. - Attention biases, one per CLIP head and shared across the recognition blocks. A per-layer bias gave no gain [arxiv:2302.12242 eq. 2] [arxiv:2302.12242 tab. 11]. The biases are added to the attention of CLIP's last blocks. Copies of the [CLS] token, called [SLS] tokens, read the visual tokens under that bias [arxiv:2302.12242 eq. 3]. Their outputs are compared with CLIP text embeddings of the class names, and the mask proposals and class scores are combined into a map [arxiv:2302.12242 eq. 4] [arxiv:2302.12242 §3]. The released code's inference differs from eq. 4; see Contradictions. Training is end-to-end. The classification loss passes through CLIP's recognition blocks into the side network, so the proposals are learned to suit CLIP [arxiv:2302.12242 §3] [arxiv:2302.12242 §4.4]. On ADE-150 with ViT-B/16, end-to-end training scores 26.1 against 21.6 for two-stage training [arxiv:2302.12242 tab. 6]. The model is trained on COCO-Stuff's 171 classes for 60K iterations at batch size 32 [arxiv:2302.12242 §4.1] [arxiv:2302.12242 §4.2]. Only CLIP's positional embedding is trained; the other CLIP weights stay frozen [arxiv:2302.12242 §4.4] [arxiv:2302.12242 tab. 9]. Reported results (author-reported, mIoU, trained on COCO-Stuff and tested on five benchmarks) [arxiv:2302.12242 §4.1] [arxiv:2302.12242 tab. 2]: 27.5 on ADE-150 and 10.1 on ADE-847 with ViT-B/16; 32.1 on ADE-150 and 12.4 on ADE-847 with ViT-L/14. These SAN rows do not use ensembling. The paper's OVSeg rows, which are marked as ensembled, score 24.8 and 29.6 on ADE-150 [arxiv:2302.12242 tab. 2]. Averaged over the five benchmarks, SAN without ensembling is 1.8 mIoU above OVSeg with ViT-B/16 and 2.3 with ViT-L/14 [arxiv:2302.12242 §4.3]. Speed was measured on one Titan Xp at 640 x 640 input [arxiv:2302.12242 §4.3]. SAN runs at 15.2 FPS with 8.4M trainable parameters. The authors' re-implementation of OVSeg runs at 0.8 FPS with 147.2M [arxiv:2302.12242 tab. 3]. The paper's "SAN ensemble" rows are not in the public code [arxiv:2302.12242 tab. 2]. The paper does not state how the side network is initialised [arxiv:2302.12242 §4.2]. Card: [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242). ### Distilled feature field in 3D Gaussian Splatting (Feature 3DGS) Each 3D Gaussian stores an N-channel feature next to its colour. The feature is trained by distilling a 2D teacher through the same splatting that renders colour [arxiv:2312.03203 §3.1] [eq. 3] [eq. 4]. The teacher is LSeg for text (M = 512 channels) or SAM for points and boxes (M = 256) [arxiv:2312.03203 §3.2]. A speed-up module renders a smaller N and lifts it back to M with a 1x1 convolution [arxiv:2312.03203 §3.2] [app. C]. The paper's sweep shows that N matters: LSeg at N = 8 reaches 0.354 mIoU, and N = 128 reaches 0.783, with training time rising from 6:40 to 19:55 [arxiv:2312.03203 tab. A]. A text or label feature is compared with each Gaussian's feature by cosine similarity, and a softmax over the label set gives a probability per label [arxiv:2312.03203 eq. 5] [arxiv:2312.03203 eq. 6]. Reported results (author-reported, Replica, LSeg teacher, mIoU) [arxiv:2312.03203 tab. 2]: 0.782 at 14.55 FPS with N = 128, 0.787 at 6.84 FPS without the speed-up, and 0.636 for NeRF-DFF at 5.38 FPS. The paper's quantitative results are all on Replica; LLFF and LERF appear only as figures [arxiv:2312.03203 §4]. Card: [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203). ### Zero-shot pixel features fused into a SLAM map (ConceptFusion) ConceptFusion builds an open-set 3D map from posed RGB-D frames. It uses the same surface-fusion pipeline as depth and colour [arxiv:2302.07241 §3.1]. Each pixel's feature mixes one embedding of the whole frame with embeddings of mask bounding boxes. The mix is weighted by similarity to the whole-frame embedding, and no part of it is trained [arxiv:2302.07241 §3.2] [arxiv:2302.07241 eq. 3] [arxiv:2302.07241 eq. 5] [arxiv:2302.07241 eq. 6]. A pixel feature is then fused into the map point it projects to, weighted by a confidence that falls with radial distance from the camera [arxiv:2302.07241 eq. 1] [arxiv:2302.07241 eq. 2]. A query is a vector in the same space. Text uses the CLIP text encoder, images use a CLIP image embedding, audio uses AudioCLIP, and a click uses the clicked point's fused feature. Each map point gets a cosine score against the query [arxiv:2302.07241 §3.3]. The paper's quantitative tests cover text and image queries and audio, not clicks [arxiv:2302.07241 tab. 1] [arxiv:2302.07241 tab. 3] [arxiv:2302.07241 tab. 4]. Reported results (author-reported, UnCoCo tabletop sequences, 3D mIoU) [arxiv:2302.07241 tab. 1]: 0.446 for structured text queries, against 0.289 for supervised OpenSeg-3D and 0.091 for zero-shot MaskCLIP-3D. On long free-form text, the paper reports 0.378 against 0.153 for OpenSeg-3D [arxiv:2302.07241 tab. 2]. For image queries it reports 0.331, against 0.134 for LSeg-3D, the best baseline there [arxiv:2302.07241 tab. 3]. For audio, it reports 64.29% accuracy on source-ambiguous queries, against 23.81% for AudioCLIP with the privileged boxes the paper gives it [arxiv:2302.07241 tab. 4]. In 2D, ConceptFusion's zero-shot numbers are mixed. On ScanNet it matches supervised OpenSeg on mAcc (0.63) and is 0.04 lower on f-mIoU. On Replica it is well below supervised OpenSeg, at 0.31 against 0.54 mAcc [arxiv:2302.07241 tab. 5]. Removing the uniqueness term of eq. 4 lowers ScanNet mAcc from 0.63 to 0.55 [arxiv:2302.07241 tab. 6]. The robot demos, a tabletop arm and a car, are qualitative. The paper reports no success rates for them [arxiv:2302.07241 §4.3]. Building the pixel features takes 10 to 15 s per image on an RTX 3090 [arxiv:2302.07241 §3]. The authors list the memory cost of dense maps as their first limitation [arxiv:2302.07241 §6]. Negated and compositional queries are not supported [arxiv:2302.07241 §6]. The released extraction script differs from eq. 4 and eq. 5, and the card's Paper vs code section records the differences [arxiv:2302.07241 eq. 4]. Card: [ConceptFusion: Open-set Multimodal 3D Mapping](/papers/2302.07241). ### Sparse-view MaskCLIP features for grasping (SparseGrasp) SparseGrasp fits a 3D Gaussian Splatting scene to three calibrated RGB views. The Gaussians are initialised from DUSt3R dense points, since the authors argue that sparse COLMAP points overfit with few views [arxiv:2412.02140 §3.1]. Each mask averages its MaskCLIP patch features, and PCA reduces the result to 16 dimensions. These targets are distilled into the Gaussians with the same alpha blending as colour [arxiv:2412.02140 §3.1] [arxiv:2412.02140 §3.2] [arxiv:2412.02140 eq. 1] [arxiv:2412.02140 eq. 2] [arxiv:2412.02140 eq. 5]. The paper states 768 channels before PCA [arxiv:2412.02140 §1]. Training runs for 7,000 iterations, about 4 minutes [arxiv:2412.02140 §4] [arxiv:2412.02140 tab. 4]. Scene changes do not trigger a rebuild. Moved pixels are found by background subtraction, and the Gaussians whose features match them are translated and rotated by render-and-compare, in about 200 ms [arxiv:2412.02140 §3.3] [arxiv:2412.02140 tab. 2]. A GraspNet model retrained to read each Gaussian's centre, scale and rotation proposes the grasps [arxiv:2412.02140 §3.4]. The paper leaves several parts open. It does not state how a text instruction is matched to the Gaussian features, whether the PCA fit is per scene or global, or how many channels each Gaussian stores [arxiv:2412.02140 §3.2] [arxiv:2412.02140 §3.4]. The text-matching step cannot be rebuilt from the paper alone. Reported results (author-reported, real KINOVA Gen2 arm) [arxiv:2412.02140 tab. 1]: 55 of 70 static grasps with three views, against 4 of 70 for F3RM and 5 of 70 for LERF-TOGO with three views, and 38 and 33 with 17 views. Scene-update success is 22 of 25 at a 200 ms budget, but 1 of 25 at 50 ms and 13 of 25 at 100 ms [arxiv:2412.02140 tab. 2]. Each test is small: seven objects with 10 attempts each, and five scenes with five trials each. Card: [SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images](/papers/2412.02140). ### Class-agnostic 3D groups with text from a detector (Gaussian Grouping) Each Gaussian gets a 16-dimensional identity encoding rather than a semantic feature. Its supervision comes from SAM masks, tracked across views with DEVA, plus a 3D kNN consistency loss [arxiv:2312.00732 §3.2] [eq. 3]. Groups are anonymous IDs, so the scene itself gives no class names [arxiv:2312.00732 app. Limitation]. Text queries need Grounding DINO to pick a mask [arxiv:2312.00732 §4.3]. A complex prompt can pick the wrong mask, and the result then has no category name [arxiv:2312.00732 app. Limitation]. Grouped Gaussians support removal, recolouring and inpainting [arxiv:2312.00732 §3.3]. Author-reported LERF-Mask mIoU is 69.7 on figurines against 52.8 for LangSplat, and 77.0 on ramen [arxiv:2312.00732 tab. 3]. The method is limited to static scenes [arxiv:2312.00732 §4.4]. Card: [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](/papers/2312.00732). ## When to use which | Situation | Use | Why | |---|---|---| | Pixel labels exist for a fixed target vocabulary | A per-pixel model trained on those labels (the paper's LSeg+ baseline) | On PASCAL-Context-59 mIoU with COCO training, LSeg+ scores 46.5 and OpenSeg 42.1 [arxiv:2112.12143 fig. 5] | | Many names unseen in training (e.g. ADE20k-847) | Caption grounding (OpenSeg) or mask-adapted CLIP (OVSeg) | OpenSeg scores 6.3 vs 3.8 for LSeg+ on A-847 with COCO training [arxiv:2112.12143 fig. 5]. OVSeg reports 9.0 on A-847 with Swin-B [arxiv:2210.04150 tab. 1]. Absolute scores are low for all of them | | Only image-caption data, no masks or pixel labels | GroupViT | Trained with image-text contrastive losses only [arxiv:2202.11094 §3.2]. Zero-shot 52.3 on VOC 2012 and 22.4 on PASCAL Context [arxiv:2202.11094 tab. 5]. Background needs its own rule | | Class-agnostic masks and captions are available, and the latency budget allows CLIP on every proposal | OVSeg | Its CLIP branch runs once per proposal. The paper reports about 0.2 s per image for MaskFormer and 0.6 s for CLIP on an A5000 with Swin-B and ViT-L/14 [arxiv:2210.04150 app. A.9] | | CLIP must stay frozen, or only prompts may be trained | OVSeg with mask prompt tuning only | MPT only scores 8.4, 26.5 and 55.4 on A-847, A-150 and PC-59. The default, fine-tuning then MPT, scores 9.0, 29.6 and 55.7 [arxiv:2210.04150 tab. 3] [arxiv:2210.04150 app. A.5]. MPT-only starts from the original CLIP weights [arxiv:2210.04150 §4.1] | | Segmentation labels exist only for seen classes, no captions are available, and a two-stage model with frozen CLIP crops is acceptable | ZSSeg | Trained on seen classes only, with no caption mining. 37.8 hIoU on the COCO-Stuff zero-shot split against 20.9 for FCN [arxiv:2112.14757 tab. 2]. Each proposal needs its own CLIP pass on a 224 x 224 crop, and MaskFormer uses 100 queries [arxiv:2112.14757 §4.2.1] [arxiv:2112.14757 §6.2] | | CLIP's weights stay frozen apart from a small trainable head, and COCO-Stuff-style training labels are available | SAN | 8.4M trainable parameters, and 27.5 on ADE-150 with ViT-B/16 against 24.8 for the paper's ensembled OVSeg row [arxiv:2302.12242 tab. 2] [arxiv:2302.12242 tab. 3]. The side network is trained from scratch for 60K iterations at batch 32 [arxiv:2302.12242 §4.2] | | Per-image latency matters, and CLIP cannot run once per proposal | SAN over two-stage OVSeg | SAN runs CLIP once per image [arxiv:2302.12242 §3]. The authors measured 15.2 FPS for SAN and 0.8 FPS for their re-implementation of OVSeg at 640 x 640 [arxiv:2302.12242 tab. 3] | | Open vocabulary is needed, and CLIP is the classifier | Do not fine-tune CLIP on a fixed label set | Fine-tuning on COCO-Stuff's 171 ground-truth classes lowers A-847 from 7.3 to 5.3 [arxiv:2210.04150 tab. 2]. The paper assumes that caption vocabularies are more diverse and keep CLIP's generalisation [arxiv:2210.04150 §1] | | Names are reworded or synonyms are needed | Several names per class, taking the maximum score | OpenSeg reports a 2.6 mIoU average gain from ensembling, using hand-made lists [arxiv:2112.12143 §4.3] [arxiv:2112.12143 app. I]. Hand-made lists are part of the gain, so reproduce them when comparing | | Backbone size is a constraint | Smaller backbone costs accuracy | ResNet-101 reaches 63.8 on PASCAL-20 with COCO and Localized Narratives vs 70.2 for EfficientNet-B7 [arxiv:2112.12143 tab. 2] | | The output needs a background class | Plan a rule yourself | OpenSeg has no "no object" class, so proposals may cover unlabelled regions [arxiv:2112.12143 §1]. GroupViT thresholds its softmax scores [arxiv:2202.11094 §4.1]. OVSeg does not give a crop threshold [arxiv:2210.04150 §3.1] | | Concepts with no matching mask proposal | Not covered by OpenSeg | The authors note the model cannot segment concepts without a segmentation proposal [arxiv:2112.12143 app. B] | | A reconstructed 3D Gaussian Splatting scene must answer text queries from any view | Feature 3DGS | Its feature field is queried by cosine similarity [arxiv:2312.03203 eq. 5]. On Replica it reaches 0.782 mIoU at 14.55 FPS, vs 0.636 for NeRF-DFF at 5.38 FPS [arxiv:2312.03203 tab. 2]. The speed-up trades quality for render cost, and N = 128 is the speed-up setting used for the main results [arxiv:2312.03203 §4] [arxiv:2312.03203 tab. A] | | The scene is static, and objects must be deleted, moved or recoloured by ID | Gaussian Grouping | Groups are selected by ID and edited without retraining the scene [arxiv:2312.00732 §3.3]. Names need a separate text detector [arxiv:2312.00732 §4.3]. Dynamic scenes are out of scope [arxiv:2312.00732 §4.4] | | A robot builds a 3D map online from posed RGB-D frames and must answer open queries by text, image or sound, with no training | ConceptFusion | Zero-shot, using the same surface fusion as depth and colour [arxiv:2302.07241 §3.1]. 0.446 3D mIoU on structured text queries against 0.289 for supervised OpenSeg-3D [arxiv:2302.07241 tab. 1]. Dense maps are memory-heavy [arxiv:2302.07241 §6], and the pixel features take 10 to 15 s per image [arxiv:2302.07241 §3] | | Only a few calibrated RGB views are available, objects are moved between grasps, and a re-scan per move is too slow | SparseGrasp | Moved objects are re-posed in about 200 ms [arxiv:2412.02140 tab. 2]. The author-reported 55 of 70 grasps with three views compares with 38 of 70 for F3RM with 17 views [arxiv:2412.02140 tab. 1]. The update needs a 200 ms budget to reach 22 of 25, and 50 ms gives 1 of 25 [arxiv:2412.02140 tab. 2]. Adding or removing objects needs a re-initialisation [arxiv:2412.02140 §4.3]. The paper releases no code, so plan a re-implementation | ## Implementation sketch The sketch shows the inference that all three variants share: embed the names, score them against region features, and map the scores back to pixels. It follows OpenSeg's text, since it is the variant with the most detail on that path. The paper gives no code, so details the text leaves open are marked in the notes below the sketch. ```pseudocode # OpenSeg inference [arxiv:2112.12143 §3.4] [eq. 6] # Input: image (3, H, W); class_names: list of K names # Output: label map (H, W) def predict(image, class_names): # N mask proposals (sigmoid) and one feature vector per proposal masks = mask_proposals(image) # (N, H, W) regions = pool_regions(masks, image) # (N, D) # One embedding per class name; multi-word names keep every word words = text_embeddings(class_names) # (K, D) # Cosine similarity between each word and each region region_logits = cosine(words, regions) # (K, N) # Per-pixel logits: region logits times masks, summed over regions pixel_logits = region_logits @ masks.flatten(1) # (K, H*W) # Argmax over the class dimension return argmax(pixel_logits, axis=K).reshape(H, W) ``` Where the variants differ: - GroupViT replaces the mask proposals with segments from its grouping stages. Its label scores are a softmax over the labels for each segment, with a background threshold [arxiv:2202.11094 §3.3] [arxiv:2202.11094 §4.1]. - OVSeg scores each proposal by combining the mask classifier with a CLIP classification of the masked crop, using a weighted geometric mean [arxiv:2210.04150 §3.1]. - ZSSeg scores each proposal with two class probabilities: the retrained query classifier and the frozen CLIP crop classifier. Their combination is described as an ensemble, and the paper gives neither its formula nor its weight [arxiv:2112.14757 §4.2.1]. - SAN has no separate crop classifier. Its class scores come from CLIP's last blocks, read by [SLS] tokens under the side network's attention biases, and its map is the mask proposals weighted by the class scores [arxiv:2302.12242 eq. 3] [arxiv:2302.12242 eq. 4]. - SparseGrasp stores one 16-dimensional PCA feature per Gaussian and compares features by similarity. Its text-to-feature step is not stated, so the sketch above cannot be completed for it [arxiv:2412.02140 §3.2] [arxiv:2412.02140 §3.4]. Notes on the sketch: - Pixel logits can be negative, because they are cosine values times sigmoid masks. The paper does not say how low scores are handled [arxiv:2112.12143 §3.4]. - Pooling is a plain mask-weighted sum; whether it is divided by mask area is not stated [arxiv:2112.12143 §3.2]. - Training details left open: the loss weight alpha in the total loss, the optimiser and the Dice implementation [arxiv:2112.12143 §3.1] [arxiv:2112.12143 §3.2]. - Grounding scores in the paper use only the ground-truth classes of each image as queries. They are an oracle setting and not comparable with mIoU [arxiv:2112.12143 §4.3]. ## Contradictions Across papers, the reported numbers are not on a common protocol. Both sides are stated; none is resolved here. - **OpenSeg rows labelled with another backbone.** OVSeg lists the COCO-trained OpenSeg row 6.3, 9.0, 21.1, 42.1 as EfficientNet-B7 [arxiv:2210.04150 tab. 1]. OpenSeg's own figure gives those numbers for ResNet-101 [arxiv:2112.12143 fig. 5]. OpenSeg's EfficientNet-B7 row is trained on COCO and Localized Narratives, with 8.1, 11.5, 26.4 and 44.8 [arxiv:2112.12143 tab. 2]. The LSeg+ row has the same issue: OVSeg labels 3.8, 7.8, 18.0 and 46.5 as EfficientNet-B7, and OpenSeg's figure gives them for ResNet-101. - **PASCAL-20 scores.** OpenSeg reports 72.2 with EfficientNet-B7 and prompt engineering [arxiv:2112.12143 tab. 2]. OVSeg reports 94.5 with Swin-B [arxiv:2210.04150 tab. 1]. The backbones and training sets differ. The cards do not show whether the two papers handle background or classes the same way on this benchmark, so the gap should not be read as a method difference. - **PASCAL Context numbers.** GroupViT reports 22.4 with CC12M and YFCC [arxiv:2202.11094 tab. 5]. OpenSeg lists the same GroupViT row, 22.4 [arxiv:2112.12143 tab. 2]. OVSeg reports 51.7 to 55.7 on PC-59 with COCO-based training [arxiv:2210.04150 tab. 1]. The training data differs, so these are not the same experiment. - **SAN ensemble column.** SAN's headline rows do not use ensembling. The paper's OVSeg rows are marked as ensembled, and the text's average gains compare the two [arxiv:2302.12242 §4.3] [arxiv:2302.12242 tab. 2]. SAN's ensembled rows, for example 28.9 on ADE-150 with ViT-B/16, are not in the public code [arxiv:2302.12242 tab. 2]. Quote the ensemble column next to any gain. - **SAN inference formula.** The paper writes the map as a product of mask proposals and class predictions [arxiv:2302.12242 eq. 4]. The released code adds a softmax over the classes and a sigmoid on the masks before summing. The card's Paper vs code section records this difference [arxiv:2302.12242 §3] [arxiv:2302.12242 eq. 4]. - **Metrics and splits.** ZSSeg reports hIoU on a zero-shot split of COCO-Stuff with 15 unseen classes [arxiv:2112.14757 tab. 2]. OVSeg and SAN report mIoU on ADE20K, Pascal Context and VOC after training on COCO-Stuff's 171 classes [arxiv:2210.04150 tab. 1] [arxiv:2302.12242 tab. 2]. The 37.8 and 39.7 numbers are therefore not directly comparable with the OVSeg and SAN rows. - **ZSSeg self-training.** The paper reports 41.5 hIoU on COCO-Stuff and 79.3 on VOC with self-training, but does not say how pseudo-labels are made [arxiv:2112.14757 §6.4] [arxiv:2112.14757 tab. 2] [arxiv:2112.14757 tab. 3]. Quote these only with that caveat. - **Consistent cases.** The OpenSeg table's GroupViT row (52.3 on PASCAL-20, 22.4 on PC-59) matches GroupViT's own table [arxiv:2112.12143 tab. 2] [arxiv:2202.11094 tab. 5]. SAN's OVSeg ViT-L/14 row (9.0 on ADE-847, 29.6 on ADE-150, 55.7 on PC-59, 94.5 on VOC) matches OVSeg's own Swin-B table [arxiv:2302.12242 tab. 2] [arxiv:2210.04150 tab. 1]. Inconsistencies inside a single paper: - **ConceptFusion margin.** The abstract claims an outperformance of "more than 40% margin on 3D IoU" over supervised baselines [arxiv:2302.07241 §1]. Table 1 gives 0.446 against 0.289 mIoU, a relative gain of about 54% by calculation. The paper does not define the 40% figure [arxiv:2302.07241 tab. 1]. - **OpenSeg keep probability.** The main text uses 0.5 [arxiv:2112.12143 §4.1] and the appendix says 0.75 [arxiv:2112.12143 app. H]. The table row for 0.75 matches the headline COCO and Localized Narratives result, which the text does not explain. - **OpenSeg backbone initialisation.** The text says NoisyStudent is worse on all benchmarks by less than 0.5 mIoU [arxiv:2112.12143 §4.4]. In the table, it is 1.0 mIoU better on PC-59 [arxiv:2112.12143 tab. 3]. - **OpenSeg batch scope.** The synchronised loss is better on three of four benchmarks, but 1.1 worse on PC-59 [arxiv:2112.12143 app. F]. The text says the strongest model uses it [arxiv:2112.12143 §4.3]. - **SAN fusion gain.** The text says multi-layer fusion adds 0.8 mIoU over single-layer fusion [arxiv:2302.12242 §4.4]. Table 4 gives 27.8 for the four-layer fusion and 27.1 for the best single layer, a gap of 0.7 [arxiv:2302.12242 tab. 4]. - **GroupViT multi-label gain.** The text says the multi-label loss improves results by 13.1% and 2.6% for hard and soft assignment, respectively. Table 1 shows the reverse: +13.1 without hard assignment and +2.6 with it [arxiv:2202.11094 §4.2] [arxiv:2202.11094 tab. 1]. - **SparseGrasp mask model.** Section 3.1 and the Figure 2 caption name FastSAM as the mask source. Section 4 says SAM ViT-H [arxiv:2412.02140 §3.1] [arxiv:2412.02140 §4]. The paper does not say which one produced the reported numbers. - **SparseGrasp novel views.** The text says the method is strong on novel views [arxiv:2412.02140 §4.1]. Table 3 gives 0.71 for SparseGrasp and 0.75 for F3RM with 17 views [arxiv:2412.02140 tab. 3]. - **GroupViT two-stage gain.** The text says the two-stage model improves CC12M+YFCC by about 7%. Table 3 shows +15.1 mask mIoU and +8.0 boundary mIoU [arxiv:2202.11094 §4.2] [arxiv:2202.11094 tab. 3]. - **OVSeg PC-59 score.** The text gives 55.5 for the Swin-B model, and the table gives 55.7, which matches the stated 13.6 gap over OpenSeg [arxiv:2210.04150 §4.2] [arxiv:2210.04150 tab. 1]. - **Feature 3DGS text encoder.** The paper says LSeg uses a CLIP ViT-L/16 text encoder [arxiv:2312.03203 §4] [arxiv:2312.03203 app. C]. The editing section uses CLIP ViT-B/32 [arxiv:2312.03203 §4.3] [arxiv:2312.03203 app. E]. The released code loads ViT-B/32 for its LSeg backbone, which the card's "Paper vs code" section records. The paper is inconsistent, and the code disagrees with the stated backbone. ## Papers - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143): OpenSeg, which learns region proposals from class-agnostic masks and aligns caption words to regions, so it can use caption data for open-vocabulary segmentation. - [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757): ZSSeg, a MaskFormer trained on seen classes whose masks are labelled by frozen CLIP crops and a retrained query classifier, with no caption data. - [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094): a ViT with grouping stages trained from image-text pairs alone, which yields label-free segments that can be named with text at test time. - [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](/papers/2210.04150): OVSeg, which adapts CLIP to masked crops using nouns mined from captions and adds mask prompt tuning to a two-stage segmenter. - [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242): SAN, a light side ViT on a frozen CLIP that predicts masks and attention biases, trained end-to-end on COCO-Stuff. - [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203): distils LSeg and SAM features into 3D Gaussians, so the scene can be queried by text or by prompts. - [ConceptFusion: Open-set Multimodal 3D Mapping](/papers/2302.07241): fuses zero-shot pixel features into a SLAM map, which can be queried by text, image, audio or click. - [SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images](/papers/2412.02140): distils compressed MaskCLIP features into Gaussians from three views, and updates moved objects by render-and-compare for grasping. - [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](/papers/2312.00732): gives each 3D Gaussian an identity encoding learned from SAM masks, for class-agnostic grouping and editing.