Open-vocabulary segmentation

Overview

Open-vocabulary segmentation assigns each pixel a label from a list of class names given at test time. The list does not have to match the training labels 2112.12143 §1. The usual obstacle is data: per-pixel models such as LSeg need pixel-level class labels, which are costly and limited to the vocabulary they were trained on 2112.12143 §1. Image-level vision-language models such as CLIP and ALIGN learn from cheap image-caption pairs but give one vector per image, so they lose where things are 2112.12143 §1.

In the 2D variants, Feature 3DGS and ConceptFusion, class names and visual features live in the same embedding space. A name is embedded with a text encoder and compared with visual features, and each pixel or segment takes the name that scores highest 2112.12143 §3.4 2202.11094 §3.3 2210.04150 §3.1 2112.14757 §4.3 2302.12242 §3 2312.03203 eq. 5. The variants differ in how the visual side is built and where its supervision comes from:

• Mask proposals, caption grounding (OpenSeg): masks are learned from class-agnostic labels, and caption words are aligned to regions.
• Text-only grouping (GroupViT): no masks at all; patches are merged into segments by a caption-matching signal.
• Two-stage mask classification (OVSeg): a mask generator proposes regions and a CLIP model, adapted to masked crops, names each one.
• Two-stage with seen-class proposals (ZSSeg): a mask generator trained on seen classes only proposes regions, and a frozen CLIP names each masked crop. No captions are used.
• Side network on frozen CLIP (SAN): a light side ViT predicts masks and attention biases, and the biases steer CLIP's last blocks to classify each mask in a single forward pass.
• Feature distillation into 3D Gaussians (Feature 3DGS): a 2D teacher's features are stored per Gaussian and queried by similarity.
• Zero-shot features fused into a SLAM map (ConceptFusion): no training. Pixel features mix a whole-frame embedding with mask-region embeddings, and are fused into map points that any query type can score.
• Sparse-view distillation for grasping (SparseGrasp): MaskCLIP region features, compressed to 16 dimensions, are distilled into Gaussians from three views. The paper does not state how a text query reaches them, so it is the exception to the shared invariant as written.
• Class-agnostic 3D groups (Gaussian Grouping): groups come from SAM masks and carry no names. Text is handled by a separate detector, so this is the exception to the shared invariant.

Variants

Region proposals with caption grounding (OpenSeg)

The image is turned into a fixed set of mask proposals, each with a feature vector, instead of one vector or a dense per-pixel map 2112.12143 §3. The paper uses N = 128 proposals 2112.12143 app. E. The model is trained in three parts:

• Masks: class-agnostic COCO panoptic masks supervise the proposals with a Dice loss, where each labelled mask is matched to its best proposal 2112.12143 §3.1 2112.12143 §4.1.
• Alignment: nouns and adjectives from captions are matched to regions with a grounding loss, using a frozen BERT-Large text encoder 2112.12143 §3.2 2112.12143 §4.1.
• Caption-only data (optional): a teacher trained on masks labels Localized Narratives with pseudo masks, and a student trains on human and pseudo labels 2112.12143 §3.3.

Reported results (author-reported, zero-shot, mIoU) 2112.12143 tab. 2:

• EfficientNet-B7, COCO and Localized Narratives: 70.2 on PASCAL-20.
• Same model with prompt engineering: 72.2 on PASCAL-20 and 28.6 on ADE20k-150.
• The paper compares the 72.2 with a 52.3 LSeg figure taken from the LSeg paper, which uses a different backbone and setup 2112.12143 §4.3.

Card: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels.

Text-only grouping, no masks (GroupViT)

GroupViT needs no mask labels. Its image encoder is a ViT-S whose Transformer layers are split into grouping stages 2202.11094 §3.1 2202.11094 §4.1. Learnable group tokens are concatenated with the segment tokens. At the end of each stage, a grouping block assigns every segment to one group and merges the segments of each group into a larger segment 2202.11094 §3.1. The headline model uses 64 group tokens and then 8 2202.11094 §4.1.

• Hard assignment: the assignment is one-hot in the forward pass and uses a straight-through gradient. The paper reports that soft assignment costs a large amount of mIoU 2202.11094 §3.1 2202.11094 tab. 1.
• Training: image-text contrastive losses only, on CC12M and filtered YFCC 2202.11094 §3.2 2202.11094 §4.1. A multi-label loss adds nouns drawn from each caption, placed in short templates, as extra positives (K = 3) 2202.11094 §3.2 2202.11094 §4.1.
• Zero-shot inference: each segment takes the label with the highest normalised similarity to the prompted label embeddings 2202.11094 §3.3. Segments with a low top probability are background, using thresholds of 0.9 on VOC 2012 and 0.5 on PASCAL Context 2202.11094 §4.1.

Reported results (author-reported, zero-shot, mIoU) 2202.11094 tab. 5: 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context. The paper describes the VOC result as approaching a fully supervised DeiT row that uses class labels 2202.11094 §4.4 2202.11094 tab. 5.

Background is the weak point. On PASCAL Context, classes such as ground, road and wall are grouped correctly but named wrongly, and their IoU is about 1.5 2202.11094 app. C.3.

Card: GroupViT: Semantic Segmentation Emerges from Text Supervision.

Mask-adapted CLIP for two-stage proposals (OVSeg)

OVSeg is a two-stage model. A MaskFormer-style mask generator, trained on COCO-Stuff with 171 classes and with CLIP text embeddings as its classifier, predicts proposals with a class-probability head 2210.04150 §3.1. A parallel CLIP branch crops each proposal, masks out its background and classifies the crop. The two scores are combined by a weighted geometric mean 2210.04150 §3.1.

The CLIP branch is adapted in three steps:

• Mining pairs: run the class-agnostic mask generator on each image, extract nouns from the captions, and pair each noun with the proposal that CLIP matches best. One caption per image gives 440K pairs and 12K nouns 2210.04150 §3.2 2210.04150 tab. 2.
• Fine-tuning: only the CLIP image encoder is trained on these pairs; the text encoder stays frozen 2210.04150 §4.1.
• Mask prompt tuning: fully masked patch tokens are replaced by learnable prompts, and prompts are also added at deeper layers. This can run with CLIP otherwise frozen 2210.04150 §3.3 2210.04150 §4.3.2.

Reported results (author-reported, mIoU) 2210.04150 tab. 1: with Swin-B and CLIP ViT-L/14, trained on COCO-Stuff-171, 9.0 on A-847, 29.6 on A-150, 55.7 on PC-59 and 94.5 on PAS-20. The ablation baseline, MaskFormer Swin-B with the original CLIP, scores 7.3, 21.8 and 51.4 on A-847, A-150 and PC-59 2210.04150 tab. 2 2210.04150 tab. 3.

Card: Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.

Seen-class proposals with frozen CLIP crops (ZSSeg)

ZSSeg is two-stage, like OVSeg, but it uses no captions. A MaskFormer is trained on the seen classes only and proposes class-agnostic masks 2112.14757 §4.1 2112.14757 §6.2. Each mask is labelled by two classifiers, and the two are combined by default 2112.14757 §4.2.1 2112.14757 tab. 7:

• Frozen crop classifier: the CLIP image encoder, unchanged, classifies a crop of the masked region.
• Retrained query classifier: the MaskFormer image encoder is retrained, with the frozen CLIP text embeddings of the seen names as fixed class weights. This pushes the encoder toward CLIP's text space, so unseen names can be scored later 2112.14757 §4.2.1.

Each mask is binarised at 0.5, its bounding box is cropped, the rest is filled with a constant colour, and the crop is resized to 224 x 224 2112.14757 §4.2.1 2112.14757 app. C.1. Among the fills the paper tests, the mean colour gives the best hIoU, 18.3 against 17.2 for zero 2112.14757 tab. 12. Class prompts come either from CLIP's 80 ImageNet templates, with the best one picked on training data, or from a learned prefix of trainable vectors 2112.14757 §4.2.2. In the paper's ablation the learned prefix raises hIoU from 18.3 to 28.2 2112.14757 tab. 14. The card's Paper vs code section lists the prompt settings in the released configs.

Reported results (author-reported, hIoU, COCO-Stuff zero-shot split of 156 seen and 15 unseen classes) 2112.14757 tab. 2: 37.8, with 39.3 mIoU on seen and 36.3 on unseen classes, against 20.9 for FCN. On Pascal VOC 2012 with 15 seen and 5 unseen classes 2112.14757 tab. 3, it reports 77.5 against 39.7 for CaGNet, the best prior method without self-training. A MaskFormer trained on all 171 classes reaches 49.4 hIoU, which is the paper's supervised upper bound 2112.14757 tab. 9. Trained on COCO-Stuff with no fine-tuning, it reports 34.5, 47.7, 20.5 and 7.0 mIoU on Cityscapes, Pascal Context, ADE20K-150 and ADE20K-847, against 21.4, 28.2, 14.9 and 4.1 for FCN 2112.14757 tab. 1.

Both branches are needed. The retrained encoder alone scores 8.7 hIoU and the frozen CLIP crops alone score 28.2, against 37.8 for the ensemble 2112.14757 tab. 7. Unseen thing classes lag unseen stuff classes: the gap is 14.9 here, against 9.5 for the supervised model 2112.14757 tab. 10. The ensemble weight is not given in the paper 2112.14757 §4.2.1, and the crop expansion ratio is described but not set 2112.14757 app. C.1.

Card: A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model.

Frozen CLIP with a side network (SAN)

SAN keeps CLIP frozen and adds a light side ViT with 8 blocks, width 240 and 6 heads. It has 8.4M trainable parameters in total 2302.12242 §4.2 2302.12242 tab. 3. The side network reads CLIP's earlier features, fused at fixed depths. The default split, nine fused CLIP blocks and three recognition blocks for ViT-B/16, scores best among the splits tested 2302.12242 tab. 5.

The side network has two outputs 2302.12242 §3:

• Mask proposals, from the inner product of pixel and query features 2302.12242 eq. 1.
• Attention biases, one per CLIP head and shared across the recognition blocks. A per-layer bias gave no gain 2302.12242 eq. 2 2302.12242 tab. 11.

The biases are added to the attention of CLIP's last blocks. Copies of the [CLS] token, called [SLS] tokens, read the visual tokens under that bias 2302.12242 eq. 3. Their outputs are compared with CLIP text embeddings of the class names, and the mask proposals and class scores are combined into a map 2302.12242 eq. 4 2302.12242 §3. The released code's inference differs from eq. 4; see Contradictions.

Training is end-to-end. The classification loss passes through CLIP's recognition blocks into the side network, so the proposals are learned to suit CLIP 2302.12242 §3 2302.12242 §4.4. On ADE-150 with ViT-B/16, end-to-end training scores 26.1 against 21.6 for two-stage training 2302.12242 tab. 6. The model is trained on COCO-Stuff's 171 classes for 60K iterations at batch size 32 2302.12242 §4.1 2302.12242 §4.2. Only CLIP's positional embedding is trained; the other CLIP weights stay frozen 2302.12242 §4.4 2302.12242 tab. 9.

Reported results (author-reported, mIoU, trained on COCO-Stuff and tested on five benchmarks) 2302.12242 §4.1 2302.12242 tab. 2: 27.5 on ADE-150 and 10.1 on ADE-847 with ViT-B/16; 32.1 on ADE-150 and 12.4 on ADE-847 with ViT-L/14. These SAN rows do not use ensembling. The paper's OVSeg rows, which are marked as ensembled, score 24.8 and 29.6 on ADE-150 [arxiv:2302.12242 tab. 2]. Averaged over the five benchmarks, SAN without ensembling is 1.8 mIoU above OVSeg with ViT-B/16 and 2.3 with ViT-L/14 2302.12242 §4.3.

Speed was measured on one Titan Xp at 640 x 640 input 2302.12242 §4.3. SAN runs at 15.2 FPS with 8.4M trainable parameters. The authors' re-implementation of OVSeg runs at 0.8 FPS with 147.2M 2302.12242 tab. 3. The paper's "SAN ensemble" rows are not in the public code 2302.12242 tab. 2. The paper does not state how the side network is initialised 2302.12242 §4.2.

Card: Side Adapter Network for Open-Vocabulary Semantic Segmentation.

Distilled feature field in 3D Gaussian Splatting (Feature 3DGS)

Each 3D Gaussian stores an N-channel feature next to its colour. The feature is trained by distilling a 2D teacher through the same splatting that renders colour 2312.03203 §3.1 eq. 3 eq. 4. The teacher is LSeg for text (M = 512 channels) or SAM for points and boxes (M = 256) 2312.03203 §3.2. A speed-up module renders a smaller N and lifts it back to M with a 1x1 convolution 2312.03203 §3.2 app. C. The paper's sweep shows that N matters: LSeg at N = 8 reaches 0.354 mIoU, and N = 128 reaches 0.783, with training time rising from 6:40 to 19:55 2312.03203 tab. A.

A text or label feature is compared with each Gaussian's feature by cosine similarity, and a softmax over the label set gives a probability per label 2312.03203 eq. 5 2312.03203 eq. 6.

Reported results (author-reported, Replica, LSeg teacher, mIoU) 2312.03203 tab. 2: 0.782 at 14.55 FPS with N = 128, 0.787 at 6.84 FPS without the speed-up, and 0.636 for NeRF-DFF at 5.38 FPS. The paper's quantitative results are all on Replica; LLFF and LERF appear only as figures 2312.03203 §4.

Card: Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields.

Zero-shot pixel features fused into a SLAM map (ConceptFusion)

ConceptFusion builds an open-set 3D map from posed RGB-D frames. It uses the same surface-fusion pipeline as depth and colour 2302.07241 §3.1. Each pixel's feature mixes one embedding of the whole frame with embeddings of mask bounding boxes. The mix is weighted by similarity to the whole-frame embedding, and no part of it is trained 2302.07241 §3.2 2302.07241 eq. 3 2302.07241 eq. 5 2302.07241 eq. 6. A pixel feature is then fused into the map point it projects to, weighted by a confidence that falls with radial distance from the camera 2302.07241 eq. 1 2302.07241 eq. 2.

A query is a vector in the same space. Text uses the CLIP text encoder, images use a CLIP image embedding, audio uses AudioCLIP, and a click uses the clicked point's fused feature. Each map point gets a cosine score against the query 2302.07241 §3.3. The paper's quantitative tests cover text and image queries and audio, not clicks 2302.07241 tab. 1 2302.07241 tab. 3 2302.07241 tab. 4.

Reported results (author-reported, UnCoCo tabletop sequences, 3D mIoU) 2302.07241 tab. 1: 0.446 for structured text queries, against 0.289 for supervised OpenSeg-3D and 0.091 for zero-shot MaskCLIP-3D. On long free-form text, the paper reports 0.378 against 0.153 for OpenSeg-3D [arxiv:2302.07241 tab. 2]. For image queries it reports 0.331, against 0.134 for LSeg-3D, the best baseline there 2302.07241 tab. 3. For audio, it reports 64.29% accuracy on source-ambiguous queries, against 23.81% for AudioCLIP with the privileged boxes the paper gives it 2302.07241 tab. 4.

In 2D, ConceptFusion's zero-shot numbers are mixed. On ScanNet it matches supervised OpenSeg on mAcc (0.63) and is 0.04 lower on f-mIoU. On Replica it is well below supervised OpenSeg, at 0.31 against 0.54 mAcc [arxiv:2302.07241 tab. 5]. Removing the uniqueness term of eq. 4 lowers ScanNet mAcc from 0.63 to 0.55 2302.07241 tab. 6.

The robot demos, a tabletop arm and a car, are qualitative. The paper reports no success rates for them 2302.07241 §4.3. Building the pixel features takes 10 to 15 s per image on an RTX 3090 2302.07241 §3. The authors list the memory cost of dense maps as their first limitation [arxiv:2302.07241 §6]. Negated and compositional queries are not supported 2302.07241 §6. The released extraction script differs from eq. 4 and eq. 5, and the card's Paper vs code section records the differences 2302.07241 eq. 4.

Card: ConceptFusion: Open-set Multimodal 3D Mapping.

Sparse-view MaskCLIP features for grasping (SparseGrasp)

SparseGrasp fits a 3D Gaussian Splatting scene to three calibrated RGB views. The Gaussians are initialised from DUSt3R dense points, since the authors argue that sparse COLMAP points overfit with few views 2412.02140 §3.1. Each mask averages its MaskCLIP patch features, and PCA reduces the result to 16 dimensions. These targets are distilled into the Gaussians with the same alpha blending as colour 2412.02140 §3.1 2412.02140 §3.2 2412.02140 eq. 1 2412.02140 eq. 2 2412.02140 eq. 5. The paper states 768 channels before PCA 2412.02140 §1. Training runs for 7,000 iterations, about 4 minutes 2412.02140 §4 2412.02140 tab. 4.

Scene changes do not trigger a rebuild. Moved pixels are found by background subtraction, and the Gaussians whose features match them are translated and rotated by render-and-compare, in about 200 ms 2412.02140 §3.3 2412.02140 tab. 2. A GraspNet model retrained to read each Gaussian's centre, scale and rotation proposes the grasps 2412.02140 §3.4.

The paper leaves several parts open. It does not state how a text instruction is matched to the Gaussian features, whether the PCA fit is per scene or global, or how many channels each Gaussian stores 2412.02140 §3.2 2412.02140 §3.4. The text-matching step cannot be rebuilt from the paper alone.

Reported results (author-reported, real KINOVA Gen2 arm) 2412.02140 tab. 1: 55 of 70 static grasps with three views, against 4 of 70 for F3RM and 5 of 70 for LERF-TOGO with three views, and 38 and 33 with 17 views. Scene-update success is 22 of 25 at a 200 ms budget, but 1 of 25 at 50 ms and 13 of 25 at 100 ms 2412.02140 tab. 2. Each test is small: seven objects with 10 attempts each, and five scenes with five trials each.

Card: SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images.

Class-agnostic 3D groups with text from a detector (Gaussian Grouping)

Each Gaussian gets a 16-dimensional identity encoding rather than a semantic feature. Its supervision comes from SAM masks, tracked across views with DEVA, plus a 3D kNN consistency loss 2312.00732 §3.2 eq. 3. Groups are anonymous IDs, so the scene itself gives no class names [arxiv:2312.00732 app. Limitation]. Text queries need Grounding DINO to pick a mask [arxiv:2312.00732 §4.3]. A complex prompt can pick the wrong mask, and the result then has no category name 2312.00732 app. Limitation.

Grouped Gaussians support removal, recolouring and inpainting 2312.00732 §3.3. Author-reported LERF-Mask mIoU is 69.7 on figurines against 52.8 for LangSplat, and 77.0 on ramen 2312.00732 tab. 3. The method is limited to static scenes 2312.00732 §4.4.

Card: Gaussian Grouping: Segment and Edit Anything in 3D Scenes.

When to use which

Implementation sketch

The sketch shows the inference that all three variants share: embed the names, score them against region features, and map the scores back to pixels. It follows OpenSeg's text, since it is the variant with the most detail on that path. The paper gives no code, so details the text leaves open are marked in the notes below the sketch.

# OpenSeg inference  [arxiv:2112.12143 §3.4] [eq. 6]
# Input:  image (3, H, W); class_names: list of K names
# Output: label map (H, W)

def predict(image, class_names):
    # N mask proposals (sigmoid) and one feature vector per proposal
    masks = mask_proposals(image)              # (N, H, W)
    regions = pool_regions(masks, image)       # (N, D)
    # One embedding per class name; multi-word names keep every word
    words = text_embeddings(class_names)       # (K, D)
    # Cosine similarity between each word and each region
    region_logits = cosine(words, regions)     # (K, N)
    # Per-pixel logits: region logits times masks, summed over regions
    pixel_logits = region_logits @ masks.flatten(1)   # (K, H*W)
    # Argmax over the class dimension
    return argmax(pixel_logits, axis=K).reshape(H, W)

Where the variants differ:

• GroupViT replaces the mask proposals with segments from its grouping stages. Its label scores are a softmax over the labels for each segment, with a background threshold 2202.11094 §3.3 2202.11094 §4.1.
• OVSeg scores each proposal by combining the mask classifier with a CLIP classification of the masked crop, using a weighted geometric mean 2210.04150 §3.1.
• ZSSeg scores each proposal with two class probabilities: the retrained query classifier and the frozen CLIP crop classifier. Their combination is described as an ensemble, and the paper gives neither its formula nor its weight 2112.14757 §4.2.1.
• SAN has no separate crop classifier. Its class scores come from CLIP's last blocks, read by [SLS] tokens under the side network's attention biases, and its map is the mask proposals weighted by the class scores 2302.12242 eq. 3 2302.12242 eq. 4.
• SparseGrasp stores one 16-dimensional PCA feature per Gaussian and compares features by similarity. Its text-to-feature step is not stated, so the sketch above cannot be completed for it 2412.02140 §3.2 2412.02140 §3.4.

Notes on the sketch:

• Pixel logits can be negative, because they are cosine values times sigmoid masks. The paper does not say how low scores are handled 2112.12143 §3.4.
• Pooling is a plain mask-weighted sum; whether it is divided by mask area is not stated 2112.12143 §3.2.
• Training details left open: the loss weight alpha in the total loss, the optimiser and the Dice implementation 2112.12143 §3.1 2112.12143 §3.2.
• Grounding scores in the paper use only the ground-truth classes of each image as queries. They are an oracle setting and not comparable with mIoU 2112.12143 §4.3.

Contradictions

Across papers, the reported numbers are not on a common protocol. Both sides are stated; none is resolved here.

• OpenSeg rows labelled with another backbone. OVSeg lists the COCO-trained OpenSeg row 6.3, 9.0, 21.1, 42.1 as EfficientNet-B7 2210.04150 tab. 1. OpenSeg's own figure gives those numbers for ResNet-101 2112.12143 fig. 5. OpenSeg's EfficientNet-B7 row is trained on COCO and Localized Narratives, with 8.1, 11.5, 26.4 and 44.8 2112.12143 tab. 2. The LSeg+ row has the same issue: OVSeg labels 3.8, 7.8, 18.0 and 46.5 as EfficientNet-B7, and OpenSeg's figure gives them for ResNet-101.
• PASCAL-20 scores. OpenSeg reports 72.2 with EfficientNet-B7 and prompt engineering 2112.12143 tab. 2. OVSeg reports 94.5 with Swin-B 2210.04150 tab. 1. The backbones and training sets differ. The cards do not show whether the two papers handle background or classes the same way on this benchmark, so the gap should not be read as a method difference.
• PASCAL Context numbers. GroupViT reports 22.4 with CC12M and YFCC 2202.11094 tab. 5. OpenSeg lists the same GroupViT row, 22.4 2112.12143 tab. 2. OVSeg reports 51.7 to 55.7 on PC-59 with COCO-based training 2210.04150 tab. 1. The training data differs, so these are not the same experiment.
• SAN ensemble column. SAN's headline rows do not use ensembling. The paper's OVSeg rows are marked as ensembled, and the text's average gains compare the two 2302.12242 §4.3 2302.12242 tab. 2. SAN's ensembled rows, for example 28.9 on ADE-150 with ViT-B/16, are not in the public code 2302.12242 tab. 2. Quote the ensemble column next to any gain.
• SAN inference formula. The paper writes the map as a product of mask proposals and class predictions 2302.12242 eq. 4. The released code adds a softmax over the classes and a sigmoid on the masks before summing. The card's Paper vs code section records this difference 2302.12242 §3 2302.12242 eq. 4.
• Metrics and splits. ZSSeg reports hIoU on a zero-shot split of COCO-Stuff with 15 unseen classes 2112.14757 tab. 2. OVSeg and SAN report mIoU on ADE20K, Pascal Context and VOC after training on COCO-Stuff's 171 classes 2210.04150 tab. 1 2302.12242 tab. 2. The 37.8 and 39.7 numbers are therefore not directly comparable with the OVSeg and SAN rows.
• ZSSeg self-training. The paper reports 41.5 hIoU on COCO-Stuff and 79.3 on VOC with self-training, but does not say how pseudo-labels are made 2112.14757 §6.4 2112.14757 tab. 2 2112.14757 tab. 3. Quote these only with that caveat.
• Consistent cases. The OpenSeg table's GroupViT row (52.3 on PASCAL-20, 22.4 on PC-59) matches GroupViT's own table 2112.12143 tab. 2 2202.11094 tab. 5. SAN's OVSeg ViT-L/14 row (9.0 on ADE-847, 29.6 on ADE-150, 55.7 on PC-59, 94.5 on VOC) matches OVSeg's own Swin-B table 2302.12242 tab. 2 2210.04150 tab. 1.

Inconsistencies inside a single paper:

• ConceptFusion margin. The abstract claims an outperformance of "more than 40% margin on 3D IoU" over supervised baselines 2302.07241 §1. Table 1 gives 0.446 against 0.289 mIoU, a relative gain of about 54% by calculation. The paper does not define the 40% figure 2302.07241 tab. 1.
• OpenSeg keep probability. The main text uses 0.5 2112.12143 §4.1 and the appendix says 0.75 2112.12143 app. H. The table row for 0.75 matches the headline COCO and Localized Narratives result, which the text does not explain.
• OpenSeg backbone initialisation. The text says NoisyStudent is worse on all benchmarks by less than 0.5 mIoU 2112.12143 §4.4. In the table, it is 1.0 mIoU better on PC-59 2112.12143 tab. 3.
• OpenSeg batch scope. The synchronised loss is better on three of four benchmarks, but 1.1 worse on PC-59 2112.12143 app. F. The text says the strongest model uses it 2112.12143 §4.3.
• SAN fusion gain. The text says multi-layer fusion adds 0.8 mIoU over single-layer fusion 2302.12242 §4.4. Table 4 gives 27.8 for the four-layer fusion and 27.1 for the best single layer, a gap of 0.7 2302.12242 tab. 4.
• GroupViT multi-label gain. The text says the multi-label loss improves results by 13.1% and 2.6% for hard and soft assignment, respectively. Table 1 shows the reverse: +13.1 without hard assignment and +2.6 with it 2202.11094 §4.2 2202.11094 tab. 1.
• SparseGrasp mask model. Section 3.1 and the Figure 2 caption name FastSAM as the mask source. Section 4 says SAM ViT-H 2412.02140 §3.1 2412.02140 §4. The paper does not say which one produced the reported numbers.
• SparseGrasp novel views. The text says the method is strong on novel views 2412.02140 §4.1. Table 3 gives 0.71 for SparseGrasp and 0.75 for F3RM with 17 views 2412.02140 tab. 3.
• GroupViT two-stage gain. The text says the two-stage model improves CC12M+YFCC by about 7%. Table 3 shows +15.1 mask mIoU and +8.0 boundary mIoU 2202.11094 §4.2 2202.11094 tab. 3.
• OVSeg PC-59 score. The text gives 55.5 for the Swin-B model, and the table gives 55.7, which matches the stated 13.6 gap over OpenSeg 2210.04150 §4.2 2210.04150 tab. 1.
• Feature 3DGS text encoder. The paper says LSeg uses a CLIP ViT-L/16 text encoder 2312.03203 §4 2312.03203 app. C. The editing section uses CLIP ViT-B/32 2312.03203 §4.3 2312.03203 app. E. The released code loads ViT-B/32 for its LSeg backbone, which the card's "Paper vs code" section records. The paper is inconsistent, and the code disagrees with the stated backbone.

Papers

• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels: OpenSeg, which learns region proposals from class-agnostic masks and aligns caption words to regions, so it can use caption data for open-vocabulary segmentation.
• A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model: ZSSeg, a MaskFormer trained on seen classes whose masks are labelled by frozen CLIP crops and a retrained query classifier, with no caption data.
• GroupViT: Semantic Segmentation Emerges from Text Supervision: a ViT with grouping stages trained from image-text pairs alone, which yields label-free segments that can be named with text at test time.
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP: OVSeg, which adapts CLIP to masked crops using nouns mined from captions and adds mask prompt tuning to a two-stage segmenter.
• Side Adapter Network for Open-Vocabulary Semantic Segmentation: SAN, a light side ViT on a frozen CLIP that predicts masks and attention biases, trained end-to-end on COCO-Stuff.
• Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields: distils LSeg and SAM features into 3D Gaussians, so the scene can be queried by text or by prompts.
• ConceptFusion: Open-set Multimodal 3D Mapping: fuses zero-shot pixel features into a SLAM map, which can be queried by text, image, audio or click.
• SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images: distils compressed MaskCLIP features into Gaussians from three views, and updates moved objects by render-and-compare for grasping.
• Gaussian Grouping: Segment and Edit Anything in 3D Scenes: gives each 3D Gaussian an identity encoding learned from SAM masks, for class-agnostic grouping and editing.