Referring segmentation

Overview

Referring segmentation takes an image and a text expression and returns a mask for the object the text picks out. The two papers on this page extend the task so that a language model answers in free text, and every object it names gets its own mask. Earlier referring models output one object per query and cannot hold a conversation 2311.03356 §1.

PixelLM takes the multi-target case: its answer can name several objects, each with a mask. Its benchmarks are the refCOCO family for single referred objects, a multi-referring split, and the authors' MUSE set for multi-target answers 2312.02228 §5.2 2312.02228 §4.1. GLaMM takes grounded conversation: a caption in which each phrase carries a mask, and user boxes can be given as input 2311.03356 §3.1 2311.03356 §3.2. It introduces the GCG benchmark for that setting 2311.03356 §3.2.

The shared idea is that the language model does not output pixels. It emits special tokens, and a decoder reads their hidden states together with image features to produce each mask 2312.02228 §3.1 2311.03356 §3.1. The two designs differ in what the tokens are and where the image features come from. Earlier systems such as LISA also use the language model for the answer but hand the mask step to a separate SAM model, which costs compute and caps quality at SAM's level 2312.02228 §1. PixelLM avoids that dependency. GLaMM keeps SAM's image encoder and a SAM-style decoder 2311.03356 §3.1 2311.03356 app. A.2.

Variants

Learned codebook tokens with a pixel decoder (PixelLM)

A LLaVA-style model with LoRA tuning writes, for each target, a group of learned tokens. The paper uses two feature scales with three tokens per scale, six codebook tokens per target 2312.02228 §3.1 2312.02228 §5.5. The hidden states of each group are fused per scale, then a pixel decoder produces one mask per target from the deepest scale upward [arxiv:2312.02228 §3.1] 2312.02228 eq. 1. A refinement loss, a BCE term and a DICE term train the model 2312.02228 §3.2.

Reported results, all author-reported:

• Multi-target answers on MUSE: PixelLM-7B reaches 39.2 gIoU and 46.3 cIoU on the test set, against 12.8 / 27.1 for LISA-7B and 38.9 / 44.4 for LISA-7B trained on MUSE 2312.02228 tab. 1.
• Multi-referring: PixelLM is above LISA trained on MUSE on all eight splits of tab. 2; the metric is not named in the caption 2312.02228 tab. 2.
• Referring (refCOCO family): see the comparison in "When to use which".
• Compute: 3.57 TFLOPs for the 7B model against 7.16 for LISA-7B, and 6.65 against 10.24 for 13B. The values are computed at 512 generated tokens, not measured 2312.02228 tab. 1 2312.02228 app. A.

Card: PixelLM: Pixel Reasoning with Large Multimodal Model.

One segment token per object, SAM-based decoder, grounded chat (GLaMM)

A CLIP image encoder and a 7B Vicuna LLM with LoRA produce the answer. The model writes a <SEG> token wherever it names an object, and the last-layer state at that token is passed to a pixel decoder with SAM-style design, which reads SAM image features 2311.03356 §3.1. The grounding image encoder is SAM ViT-H, frozen, at 1024 px 2311.03356 app. A.2 train.py. A region encoder turns user boxes into features that replace <bbox> slots in the prompt 2311.03356 §3.1.

The GCG output asks for a caption in which each grounded phrase is wrapped in tags and followed by <SEG> 2311.03356 §3.2. Training uses one text cross-entropy term and per-mask BCE and DICE terms 2311.03356 app. A.2.

Reported results, all author-reported:

• GCG, test split: GLaMM reaches 65.6 mIoU and 43.5 CIDEr, against 61.7 and 32.2 for LISA, which the authors adapted to GCG themselves [arxiv:2311.03356 tab. 3].
• Referring segmentation, refCOCO val: 79.5 for GLaMM against 74.9 for LISA-7B. The table caption calls the assessment "qualitative" and names no metric 2311.03356 tab. 4.
• Compute: no compute figures are reported in the card's source tables.

Card: GLaMM: Pixel Grounding Large Multimodal Model.

When to use which

Implementation sketch

Inference for one query, following PixelLM's §3.1 and eq. 1. Fusion and scale weights are simplified here; see Contradictions for what the official code does.

# One referring pass  [arxiv:2312.02228 §3.1] [arxiv:2312.02228 eq. 1]
# Input:  image (3, H, W); query text
# Output: one mask per target named in the answer

def refer_segment(image, query):
    # Frozen vision features at each chosen scale  [arxiv:2312.02228 §3.1]
    feats = [encode(image, layer) for layer in scale_layers]
    # The language model answers and writes N codebook tokens per target  [§3.1]
    answer, hidden = lmm.generate(image, query)
    for group in codebook_groups(answer):
        # Fuse each scale's N token states into one embedding  [§3.1]
        seg_emb = [fuse(hidden[group][l]) for l in range(L)]
        # Decode from the deepest scale upward  [eq. 1]
        masks = [None] * L
        for l in reversed(range(L)):
            f = feats[l]
            if masks[l + 1] is not None:
                # Gate this scale with the coarser scale's mask  [eq. 1]
                f = f * (sigmoid(masks[l + 1]) + 1)
            masks[l] = decode_block(seg_emb[l], f)  # m^l  [§3.1]
        # Learned weights, constrained to |gamma| = 1  [§3.1]
        yield sum(gamma[l] * masks[l] for l in range(L))

Points the text leaves open: how many scales and which layers (the paper gives 2 and layers 23 and 14, 1-based 2312.02228 tab. A.3), how the N tokens are fused (a linear layer φ in §3.1), and the refinement loss, which §3.2 defines with a weight α and a binary mask. Pick each explicitly and record the choice.

GLaMM differs in three places 2311.03356 §3.1. It emits one <SEG> token per object instead of N codebook tokens, so there is no multi-scale fusion step. Its decoder reads SAM image features at 1024 px, not CLIP feature scales. Its prompt can also carry a region vector per user box, filled into a <bbox> slot before the LLM runs.

Contradictions

• LISA numbers on refCOCO. PixelLM reports LISA at 74.1 on val and 71.1 on testB 2312.02228 tab. 3. GLaMM reports LISA-7B at 74.9 on val and 72.3 on testB 2311.03356 tab. 4. Neither caption names the metric, and this page cannot say which setup is right.
• Referring results. The PixelLM text says PixelLM is best on most metrics 2312.02228 §5.4. Tab. 3 shows LISA higher on refCOCO val and testB 2312.02228 tab. 3. The active tab. 3 caption states no metric; a commented-out caption in the source calls it cIoU 2312.02228 tab. 3.
• Benchmarks differ. The headline numbers of the two papers come from different benchmarks (MUSE and multi-referring for PixelLM; GCG for GLaMM), so the papers do not directly show which design scores higher.
• Tokens per group. Fig. 2's caption says two tokens per scale 2312.02228 fig. 2. Section 5.5 uses three 2312.02228 §5.5.
• Refinement loss. §3.2 defines it with α = 2.0 2312.02228 tab. A.1. The official code defines the function but does not call it, and the training sum is BCE plus DICE only PixelLM.py PixelLM.py.
• Token fusion. §3.1 uses a linear layer φ 2312.02228 §3.1. The code combines the tokens with one scalar per token PixelLM.py PixelLM.py. Those scalars sit in a plain Python list, so PyTorch may not train them. Check a checkpoint before relying on either.
• Scale weights. §3.1 learns γ under |γ| = 1, and the code uses a list initialised to 1/L PixelLM.py PixelLM.py.
• Gradient clipping. Appendix A says the authors do not use gradient clipping 2312.02228 app. A. The DeepSpeed config sets 1.0 train_ds.py.
• Layer for the language model. §3.1 says the final layer, and tab. A.3 calls the baseline the penultimate layer 2312.02228 §3.1 2312.02228 tab. A.3.
• Ablation text. §5.5 gives "up to 2.4%" gain from token fusion on MUSE val cIoU; tab. 4 shows 48.0 to 50.1, a gain of 2.1 2312.02228 §5.5 2312.02228 tab. 4.
• GLaMM vision encoder. The paper names CLIP ViT-H/14 [arxiv:2311.03356 §3.1]. The code's default is CLIP ViT-L/14-336 train.py.
• GLaMM image tokens. The paper says 256 image tokens [arxiv:2311.03356 §3.1]. The code pads 575 positions, which implies 576 [code: model/GLaMM.py#L172].
• GLaMM loss weights. The paper gives a linear combination with no weights 2311.03356 app. A.2. The code uses cross-entropy 1.0, BCE 2.0 and DICE 0.5 train.py.
• GLaMM trainable parts. The paper says the V-L projection is fully finetuned 2311.03356 app. A.2. The finetuning code freezes it train.py.
• GLaMM segment sampling for GCG. The finetune script uses 3, 9, 9, 9, 1 finetune_glamm_gcg.sh. The training docs use 1, 3, 3, 3, 1 training.md. The repo disagrees with itself.

Papers

• PixelLM: Pixel Reasoning with Large Multimodal Model: codebook tokens and a pixel decoder for multi-target masks, without SAM, with MUSE and referring benchmarks.
• GLaMM: Pixel Grounding Large Multimodal Model: one segment token per object in grounded chat, optional user boxes, a SAM-based decoder, and the GCG benchmark.