---
kind: concept
slug: referring-segmentation
title: Referring segmentation
summary: >-
  Output a mask for each object a text query or chat answer names, in one model. PixelLM writes learned codebook tokens and a small decoder turns them into masks, with no separate SAM model. GLaMM writes one segment token per object in a grounded chat, decodes it with a SAM-based decoder and accepts user boxes. Both report gains over LISA, author-reported and mostly on different benchmarks.
tags: [segmentation, multimodal-llm, referring-segmentation, vision-language]
papers: ["2311.03356", "2312.02228"]
related: [multimodal-language-model, visual-grounding, semantic-segmentation, open-vocabulary-segmentation, segmentation-evaluation, low-rank-adaptation]
status: draft
updated_at: 2026-10-19
---

## Overview

Referring segmentation takes an image and a text expression and returns a mask
for the object the text picks out. The two papers on this page extend the task
so that a language model answers in free text, and every object it names gets
its own mask. Earlier referring models output one object per query and cannot
hold a conversation [arxiv:2311.03356 §1].

PixelLM takes the multi-target case: its answer can name several objects, each
with a mask. Its benchmarks are the refCOCO family for single referred objects,
a multi-referring split, and the authors' MUSE set for multi-target answers
[arxiv:2312.02228 §5.2] [arxiv:2312.02228 §4.1]. GLaMM takes grounded
conversation: a caption in which each phrase carries a mask, and user boxes can
be given as input [arxiv:2311.03356 §3.1] [arxiv:2311.03356 §3.2]. It introduces
the GCG benchmark for that setting [arxiv:2311.03356 §3.2].

The shared idea is that the language model does not output pixels. It emits
special tokens, and a decoder reads their hidden states together with image
features to produce each mask [arxiv:2312.02228 §3.1] [arxiv:2311.03356 §3.1].
The two designs differ in what the tokens are and where the image features
come from. Earlier systems such as LISA also use the language model for the
answer but hand the mask step to a separate SAM model, which costs compute and
caps quality at SAM's level [arxiv:2312.02228 §1]. PixelLM avoids that
dependency. GLaMM keeps SAM's image encoder and a SAM-style decoder
[arxiv:2311.03356 §3.1] [arxiv:2311.03356 app. A.2].

## Variants

### Learned codebook tokens with a pixel decoder (PixelLM)

A LLaVA-style model with LoRA tuning writes, for each target, a group of
learned tokens. The paper uses two feature scales with three tokens per scale,
six codebook tokens per target [arxiv:2312.02228 §3.1] [arxiv:2312.02228 §5.5].
The hidden states of each group are fused per scale, then a pixel decoder
produces one mask per target from the deepest scale upward [arxiv:2312.02228
§3.1] [arxiv:2312.02228 eq. 1]. A refinement loss, a BCE term and a DICE term
train the model [arxiv:2312.02228 §3.2].

Reported results, all author-reported:

- Multi-target answers on MUSE: PixelLM-7B reaches 39.2 gIoU and 46.3 cIoU on
  the test set, against 12.8 / 27.1 for LISA-7B and 38.9 / 44.4 for LISA-7B
  trained on MUSE [arxiv:2312.02228 tab. 1].
- Multi-referring: PixelLM is above LISA trained on MUSE on all eight splits
  of tab. 2; the metric is not named in the caption [arxiv:2312.02228 tab. 2].
- Referring (refCOCO family): see the comparison in "When to use which".
- Compute: 3.57 TFLOPs for the 7B model against 7.16 for LISA-7B, and 6.65
  against 10.24 for 13B. The values are computed at 512 generated tokens, not
  measured [arxiv:2312.02228 tab. 1] [arxiv:2312.02228 app. A].

Card: [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228).

### One segment token per object, SAM-based decoder, grounded chat (GLaMM)

A CLIP image encoder and a 7B Vicuna LLM with LoRA produce the answer. The
model writes a `<SEG>` token wherever it names an object, and the last-layer
state at that token is passed to a pixel decoder with SAM-style design, which
reads SAM image features [arxiv:2311.03356 §3.1]. The grounding image encoder
is SAM ViT-H, frozen, at 1024 px [arxiv:2311.03356 app. A.2]
[code: train.py#L47]. A region encoder turns user boxes into features that
replace `<bbox>` slots in the prompt [arxiv:2311.03356 §3.1].

The GCG output asks for a caption in which each grounded phrase is wrapped in
tags and followed by `<SEG>` [arxiv:2311.03356 §3.2]. Training uses one text
cross-entropy term and per-mask BCE and DICE terms [arxiv:2311.03356 app. A.2].

Reported results, all author-reported:

- GCG, test split: GLaMM reaches 65.6 mIoU and 43.5 CIDEr, against 61.7 and
  32.2 for LISA, which the authors adapted to GCG themselves [arxiv:2311.03356
  tab. 3].
- Referring segmentation, refCOCO val: 79.5 for GLaMM against 74.9 for
  LISA-7B. The table caption calls the assessment "qualitative" and names no
  metric [arxiv:2311.03356 tab. 4].
- Compute: no compute figures are reported in the card's source tables.

Card: [GLaMM: Pixel Grounding Large Multimodal Model](/papers/2311.03356).

## When to use which

| Situation | Use | Why |
|---|---|---|
| The stack must not depend on a separate SAM-class model | The codebook-token decoder | PixelLM is marked as needing no SAM [arxiv:2312.02228 tab. 1] [arxiv:2312.02228 tab. 3]. GLaMM keeps a frozen SAM ViT-H encoder and a SAM-style decoder [arxiv:2311.03356 app. A.2] |
| One answer names several objects | Either; check the benchmark | PixelLM's multi-target numbers are on MUSE and multi-referring [arxiv:2312.02228 tab. 1] [arxiv:2312.02228 tab. 2]. GLaMM's are on GCG, where each phrase has a mask [arxiv:2311.03356 tab. 3]. The two are not comparable |
| The user points at regions with boxes | GLaMM | GLaMM takes boxes through a region encoder [arxiv:2311.03356 §3.1]. This card does not show PixelLM taking boxes |
| Output is a chat caption with a mask per phrase | GLaMM | This is the GCG task it introduces [arxiv:2311.03356 §3.2] |
| Inference compute is tight | The codebook-token decoder, if the TFLOPs figure matters | PixelLM's reported TFLOPs are about half of LISA's at the same model size [arxiv:2312.02228 tab. 1]. The figure is computed, not measured. GLaMM's card reports no compute figures; its grounding encoder runs once per image, one batch item at a time [code: model/GLaMM.py#L122-L128] [code: model/GLaMM.py#L176-L191] |
| Single-referent benchmarks are the target metric | Check per split | On refCOCO, PixelLM scores 73.0 on val and 68.2 on testB, against 74.1 and 71.1 for LISA [arxiv:2312.02228 tab. 3]. GLaMM scores 79.5 on val and 76.9 on testB, against 74.9 and 72.3 for LISA-7B [arxiv:2311.03356 tab. 4]. Neither table names a metric, and the LISA numbers differ between the two papers (see Contradictions) |
| The model must be trained from a small budget | Plan for the stated budget | PixelLM: 8 A100 GPUs for about 1.5 days (7B) or 2 days (13B) [arxiv:2312.02228 §5.1]. GLaMM: 8 A100-40GB for finetuning [arxiv:2311.03356 app. A.3]; its GranD pretraining code is not in the repo [code: GranD/run_pipeline.sh#L18] |
| The MUSE training data must be rebuilt | Budget for GPT-4V access | The data was generated with GPT-4V and then filtered with GPT-4V and human checks [arxiv:2312.02228 §4.2] [arxiv:2312.02228 app. C.1] |
| Training data must be cleared for commercial use | Check upstream terms first | GLaMM's GranD_f repurposes Flickr30k, RefCOCOg and PSG with GPT-4 captions, and GranD is built on SA-1B images with open models. The card did not check these terms [arxiv:2311.03356 §4.5] [arxiv:2311.03356 app. A.4] |

## Implementation sketch

Inference for one query, following PixelLM's §3.1 and eq. 1. Fusion and
scale weights are simplified here; see Contradictions for what the official
code does.

```pseudocode
# One referring pass  [arxiv:2312.02228 §3.1] [arxiv:2312.02228 eq. 1]
# Input:  image (3, H, W); query text
# Output: one mask per target named in the answer

def refer_segment(image, query):
    # Frozen vision features at each chosen scale  [arxiv:2312.02228 §3.1]
    feats = [encode(image, layer) for layer in scale_layers]
    # The language model answers and writes N codebook tokens per target  [§3.1]
    answer, hidden = lmm.generate(image, query)
    for group in codebook_groups(answer):
        # Fuse each scale's N token states into one embedding  [§3.1]
        seg_emb = [fuse(hidden[group][l]) for l in range(L)]
        # Decode from the deepest scale upward  [eq. 1]
        masks = [None] * L
        for l in reversed(range(L)):
            f = feats[l]
            if masks[l + 1] is not None:
                # Gate this scale with the coarser scale's mask  [eq. 1]
                f = f * (sigmoid(masks[l + 1]) + 1)
            masks[l] = decode_block(seg_emb[l], f)  # m^l  [§3.1]
        # Learned weights, constrained to |gamma| = 1  [§3.1]
        yield sum(gamma[l] * masks[l] for l in range(L))
```

Points the text leaves open: how many scales and which layers (the paper
gives 2 and layers 23 and 14, 1-based [arxiv:2312.02228 tab. A.3]), how the
N tokens are fused (a linear layer φ in §3.1), and the refinement loss, which
§3.2 defines with a weight α and a binary mask. Pick each explicitly and
record the choice.

GLaMM differs in three places [arxiv:2311.03356 §3.1]. It emits one `<SEG>`
token per object instead of N codebook tokens, so there is no multi-scale
fusion step. Its decoder reads SAM image features at 1024 px, not CLIP feature
scales. Its prompt can also carry a region vector per user box, filled into a
`<bbox>` slot before the LLM runs.

## Contradictions

- **LISA numbers on refCOCO.** PixelLM reports LISA at 74.1 on val and 71.1 on
  testB [arxiv:2312.02228 tab. 3]. GLaMM reports LISA-7B at 74.9 on val and
  72.3 on testB [arxiv:2311.03356 tab. 4]. Neither caption names the metric,
  and this page cannot say which setup is right.
- **Referring results.** The PixelLM text says PixelLM is best on most metrics
  [arxiv:2312.02228 §5.4]. Tab. 3 shows LISA higher on refCOCO val and testB
  [arxiv:2312.02228 tab. 3]. The active tab. 3 caption states no metric; a
  commented-out caption in the source calls it cIoU [arxiv:2312.02228 tab. 3].
- **Benchmarks differ.** The headline numbers of the two papers come from
  different benchmarks (MUSE and multi-referring for PixelLM; GCG for GLaMM),
  so the papers do not directly show which design scores higher.
- **Tokens per group.** Fig. 2's caption says two tokens per scale
  [arxiv:2312.02228 fig. 2]. Section 5.5 uses three [arxiv:2312.02228 §5.5].
- **Refinement loss.** §3.2 defines it with α = 2.0 [arxiv:2312.02228 tab. A.1].
  The official code defines the function but does not call it, and the
  training sum is BCE plus DICE only [code: model/PixelLM.py#L601-L602]
  [code: model/PixelLM.py#L609-L617].
- **Token fusion.** §3.1 uses a linear layer φ [arxiv:2312.02228 §3.1]. The
  code combines the tokens with one scalar per token [code: model/PixelLM.py#L287]
  [code: model/PixelLM.py#L455-L458]. Those scalars sit in a plain Python list,
  so PyTorch may not train them. Check a checkpoint before relying on either.
- **Scale weights.** §3.1 learns γ under |γ| = 1, and the code uses a list
  initialised to 1/L [code: model/PixelLM.py#L290] [code: model/PixelLM.py#L512-L514].
- **Gradient clipping.** Appendix A says the authors do not use gradient
  clipping [arxiv:2312.02228 app. A]. The DeepSpeed config sets 1.0
  [code: train_ds.py#L418].
- **Layer for the language model.** §3.1 says the final layer, and tab. A.3
  calls the baseline the penultimate layer [arxiv:2312.02228 §3.1]
  [arxiv:2312.02228 tab. A.3].
- **Ablation text.** §5.5 gives "up to 2.4%" gain from token fusion on MUSE
  val cIoU; tab. 4 shows 48.0 to 50.1, a gain of 2.1 [arxiv:2312.02228 §5.5]
  [arxiv:2312.02228 tab. 4].
- **GLaMM vision encoder.** The paper names CLIP ViT-H/14 [arxiv:2311.03356
  §3.1]. The code's default is CLIP ViT-L/14-336 [code: train.py#L41].
- **GLaMM image tokens.** The paper says 256 image tokens [arxiv:2311.03356
  §3.1]. The code pads 575 positions, which implies 576 [code:
  model/GLaMM.py#L172].
- **GLaMM loss weights.** The paper gives a linear combination with no weights
  [arxiv:2311.03356 app. A.2]. The code uses cross-entropy 1.0, BCE 2.0 and
  DICE 0.5 [code: train.py#L92-L94].
- **GLaMM trainable parts.** The paper says the V-L projection is fully
  finetuned [arxiv:2311.03356 app. A.2]. The finetuning code freezes it
  [code: train.py#L212-L215].
- **GLaMM segment sampling for GCG.** The finetune script uses 3, 9, 9, 9, 1
  [code: scripts/finetune_glamm_gcg.sh#L39]. The training docs use 1, 3, 3, 3, 1
  [code: docs/training.md#L23]. The repo disagrees with itself.

## Papers

- [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228):
  codebook tokens and a pixel decoder for multi-target masks, without SAM,
  with MUSE and referring benchmarks.
- [GLaMM: Pixel Grounding Large Multimodal Model](/papers/2311.03356): one
  segment token per object in grounded chat, optional user boxes, a SAM-based
  decoder, and the GCG benchmark.