GLaMM: Pixel Grounding Large Multimodal Model

In plain words

Chat models that look at pictures can describe them, but they usually cannot point at the things they mention. GLaMM is a picture-chat model that does: when its answer names an object, it can also return an outline of that object in the image. You can ask about the whole picture, or about a region you mark with a box, and the answer can mix words and outlines.

To teach this, the authors built a large dataset of pictures in which almost every described object already has an outline, and they wrote a new benchmark where the answer is a caption with each phrase tied to its outline. On these tests the model is compared with earlier systems, and the numbers reported are the authors' own.

TL;DR

GLaMM couples a CLIP image encoder and a 7B LLM (Vicuna, LoRA-tuned) with a region encoder for user boxes and a SAM-based pixel decoder §3.1. Special tokens mark where a region or a mask goes in the text. The paper introduces Grounded Conversation Generation (GCG), a benchmark of captions with a mask per phrase, and the GranD dataset, with 7.5M concepts grounded in 810M masks §4 tab. 2. Reported results beat LISA on GCG and on refCOCO-family segmentation tab. 3 tab. 4. All numbers are author-reported.

Problem

Earlier large multimodal models answer in text but ground little. Region-aware models take user boxes or output boxes, but do not give dense pixel-level masks inside a conversation §1 tab. 1. Referring segmentation models output one object per query and cannot hold a conversation §1. The paper also notes that no standard benchmark existed for a conversation whose phrases are each tied to masks, and that dense region labels are costly to collect by hand §1 §4. The task is therefore: generate a fluent answer in which each named object carries a segmentation mask, and accept text, boxes or both as input.

Method

Architecture §3.1 fig. 2. Five parts. (i) A global image encoder, a CLIP ViT, gives scene features. (ii) A region encoder builds region features from user boxes. (iii) A Vicuna-based LLM writes the answer. (iv) A grounding image encoder, the SAM image encoder, produces dense features at 1024 px train.py. (v) A pixel decoder with a SAM-decoder-like design turns a text embedding into a mask. A vision-to-language projection (f) maps image features into the LLM, and a language-to-prompt projection (g) maps <SEG> states into the decoder's prompt space §3.1.

Scene level §3.1. The global image features are projected into the LLM and replace the <image> token. The paper states 256 tokens §3.1; the code pads with 575 positions before the <SEG> mask, which implies 576 image positions GLaMM.py. See Paper vs code.

Region level §3.1 layers.py. Four CLIP hidden states are taken from the global encoder and fused into a feature pyramid. A RoIAlign step then pools each user box to a 14×14 grid, which is combined into one region vector. That vector replaces a <bbox> token in the prompt, and the same vision-to-language projection maps it into the LLM §3.1. The paper does not name the four CLIP layers; the code uses hidden states at offsets −11, −8, −5 and −2 llava_with_region_arch.py.

Pixel level §3.1. The LLM can emit a <SEG> token wherever it names an object. The last-layer hidden state at each <SEG> goes through the language-to-prompt projection and into the pixel decoder with the SAM image features. The decoder outputs one mask per <SEG>. The paper writes masks as binary values §3.1, while the training loss uses logits [code: model/GLaMM.py#L27-L34]; at evaluation the logits are thresholded at 0 [code: eval/gcg/infer.py#L168].

GCG output format §3.2. The answer is a caption in which each grounded phrase is wrapped in <p> and </p> and followed by <SEG>. The prompt asks for a detailed description with interleaved masks §3.2.

Pretraining and finetuning app. A.2 app. A.3. The paper pretrains on GranD for four tasks at once: referring segmentation, region captioning, image captioning and GCG app. A.2. It then finetunes per task. GCG is finetuned on GranD_f, referring segmentation on refCOCO, refCOCO+ and refCOCOg, region captioning on refCOCOg and Visual Genome, and image captioning on LLaVA-Instruct data app. A.3. The global and grounding image encoders are frozen; the region encoder, the projections and the pixel decoder are trained; the LLM gets LoRA, according to app. A.2. The code differs on the projector, see Paper vs code. The public repo ships only finetuning scripts [code: scripts/finetune_glamm_gcg.sh].

Losses §3.1 app. A.2 GLaMM.py. The paper gives a text cross-entropy plus a per-pixel binary cross-entropy and a DICE term, as a linear combination, without weights app. A.2. The code's weights are given below. The paper's equations are unnumbered §3.1; the pseudocode below follows the code.

# Forward pass for one turn, inference  [§3.1]
# Input:  image; prompt text; optional boxes (N, 4) in [0, 1]
# Output: generated text; one binary mask per <SEG> token

def glamm_answer(image, prompt, boxes=None):
    # Map global CLIP features into the LLM's embedding space  [§3.1]
    image_feats = clip_global_encoder(image)  # (N_img, D_v)
    image_tokens = vl_projector(image_feats)  # (N_img, D_t)
    # Fill each <bbox> slot with a projected region feature  [§3.1]
    if boxes is not None:
        region_feats = region_encoder(image, boxes)  # (N, D_v)
        prompt = fill_bbox_slots(prompt, vl_projector(region_feats))
    # The LLM writes text and <SEG> tokens  [§3.1]
    tokens, last_hidden = llm.generate(image_tokens, prompt)
    # Take the last-layer state at each <SEG> position  [§3.1]
    seg_states = last_hidden[is_seg_token(tokens)]  # (K, D_t)
    seg_prompts = lp_projector(seg_states)  # (K, 256) [code: GLaMM.py#L73-L78]
    # Decode one mask per prompt from the SAM image features  [§3.1]
    sam_feats = sam_image_encoder(image)  # 1024 px input
    masks = [mask_decoder(sam_feats, p) > 0 for p in seg_prompts]
    return tokens, masks

# Training loss for one batch  [§3.1] [code: model/GLaMM.py#L251-L286]
# Input:  text_loss (scalar); pred_logits (K, H, W); gt_masks (K, H, W)
# Output: scalar loss

def training_loss(text_loss, pred_logits, gt_masks):
    # Text cross-entropy, weight 1.0  [code: train.py#L92]
    ce_term = 1.0 * text_loss
    # Per-mask BCE with logits, pixel mean  [code: model/GLaMM.py#L27-L34]
    bce_per_mask = bce_with_logits(pred_logits, gt_masks)  # (K,)
    # Per-mask DICE on sigmoid outputs  [code: model/GLaMM.py#L10-L24]
    dice_per_mask = dice_loss(sigmoid(pred_logits), gt_masks)  # (K,)
    # BCE x2.0 and DICE x0.5, averaged over masks  [code: train.py#L93-L94]
    mask_term = (2.0 * sum(bce_per_mask) + 0.5 * sum(dice_per_mask)) / K
    return ce_term + mask_term

Hyperparameters

"Paper" means the paper's text or appendix. "Code" means the default in train.py or the finetune script. "not stated" means neither gives a value.

Results

Author-reported. Setup in each table caption and §5.

Grounded Conversation Generation tab. 3. Trained on GranD, then finetuned on GranD_f. Metrics: METEOR (M), CIDEr (C), class-agnostic mask AP50, mean IoU of paired masks, and mask recall §3.2 app. A.1. LISA was adapted to GCG by the authors. GLaMM† excludes the 1K human-annotated images.

Referring-expression segmentation tab. 4. The caption says "qualitative assessment" and does not name the metric. Values are per split.

Region-level captioning tab. 5. METEOR and CIDEr.

Zero-shot image captioning tab. 6. Metric values as reported.

Conversational QA, LLaVA-Bench app. B tab. 8. GLaMM scores 63.3 against 63.4 for LLaVA-1.5 (both Vicuna-7B); BLIP-2 38.1, InstructBLIP 60.9, Qwen-VL 63.4 and Qwen-VL-Chat 58.6 are also listed.

Phrase grounding app. B. Described qualitatively with a figure only; no numbers are given.

Paper vs code

Official code: mbzuai-oryx/groundingLMM (Apache-2.0), linked on the paper's first page, pinned at commit 1454b456cdd9ae9731915d2b79e9ff3553c3224a. Paths are relative to the repository root. The repo contains finetuning scripts and evaluation code, but no pretraining script for GranD [code: scripts/finetune_glamm_gcg.sh].

• Vision encoder. Paper: CLIP ViT-H/14 §3.1. Code default: openai/clip-vit-large-patch14-336 train.py. The region encoder asserts a 24 × 24 grid layers.py, which matches a 336-px ViT-L/14. The released checkpoint's config was not inspected.
• Image token count. Paper: 256 §3.1. Code: 576 patch positions, since the region path drops the CLS token and asserts 24 × 24 [code: model/llava/llava_with_region_arch.py#L116] layers.py, and the <SEG> mask pads 575 positions before the text [code: model/GLaMM.py#L172].
• LoRA alpha. Paper: α = 8 app. A.2. Code: lora_alpha 16, rank 8 [code: train.py#L89-L90]. The paper's "α = 8" may mean the rank; the code is unambiguous.
• Trainable parts. Paper: region encoder, V-L and L-P projections and pixel decoder are fully finetuned app. A.2. Code, finetuning path: the V-L projector (mm_projector) is frozen train.py, and only the SAM mask decoder is unfrozen, so the SAM prompt encoder stays frozen [code: model/GLaMM.py#L59-L71] train.py. The paper does not say which weights its pretraining stage froze, and the pretraining code is not in the repo, so this cannot be checked for GranD pretraining.
• DICE scaling. The code divides the intersection and union terms by 1000 GLaMM.py. The ratio is unchanged, but the epsilon is not: the effective epsilon is 1000 times the written 1e-6 [code: model/GLaMM.py#L11]. The paper gives no DICE formula app. A.2.
• Loss weights. Paper: a linear combination with no weights app. A.2. Code: CE 1.0, BCE 2.0, DICE 0.5 train.py.
• Segment sampling rates for GCG. Paper: not stated. Finetune script: 3, 9, 9, 9, 1 finetune_glamm_gcg.sh. Training docs: 1, 3, 3, 3, 1 training.md. The repo disagrees with itself.
• Batch size. Paper: 160 app. A.2 app. A.3. Code: 2 per device with 10 accumulation steps train.py, so 160 implies 8 GPUs, matching the hardware in app. A.3.
• Region-feature scaling. Normalized boxes are multiplied by 224 before RoIAlign layers.py, but the feature grid comes from a 336-px ViT-L/14 (24 × 24 patches at stride 14) layers.py layers.py. The dataset default image size is 224 [code: dataset/dataset.py#L27]. Whether boxes line up depends on the size the global encoder actually receives, which this review did not resolve.
• Mask recall. Thresholds of 0.5 for IoU and BERT similarity match the paper app. A.1 evaluate.py. The code uses the mean of BERT's last hidden states as the sentence embedding, then cosine similarity evaluate.py evaluate.py. The paper does not state the pooling.
• Level-1 box voting. Paper: a box is kept if at least two other detectors find it §4.1. This review did not find that rule in the GranD scripts it searched, so it is unverified.
• Projectors. Paper: V-L projection is a two-layer MLP with GELU app. A.2. Code: Linear, GELU, Linear llava_with_region_arch.py. This matches. The L-P projection is Linear, ReLU, Linear to 256 [code: model/GLaMM.py#L73-L78]; the paper does not give its form.

No discrepancies found in: the 14 × 14 RoI size §3.1 [code: model/layers.py#L209], the four region-feature layers (the paper says four but does not name them), the frozen SAM image encoder, AdamW with betas 0.9 and 0.95, or the 5K-iteration GCG finetune length app. A.3 [code: scripts/finetune_glamm_gcg.sh#L42].

Pitfalls

• Vision backbone is tied to 576 positions. The <SEG> mask padding GLaMM.py and GLaMM.py assume 575 padding positions, and the region encoder asserts a 24 × 24 grid [code: model/layers.py#L221-L222]. Swapping in the ViT-H/14 the paper names needs code changes, not only a new checkpoint.
• Box coordinate scale. See the region-feature scaling note above. Test box alignment on your input size before trusting region answers.
• Pretraining cannot be reproduced from this repo. The repo ships finetuning scripts and the GranD annotation pipeline run_pipeline.sh, but no GranD pretraining script. The pipeline asks for about ten conda environments run_pipeline.sh. The released pretrained checkpoint is named in the docs training.md.
• Unstated optimizer settings. The paper gives no optimizer, schedule, gradient clipping or weight decay app. A.2. The code gives values [code: train.py#L382-L391]; use them and say so.
• Ambiguous DICE and BCE weighting. The paper's text has no weights app. A.2. An implementer who uses the code's weights gets a stronger mask term than a plain sum would give.
• Metric for referring segmentation is not named. The caption calls the table "qualitative" tab. 4, so a reader cannot tell whether values are IoU-based. Do not compare them with the cIoU of other papers without checking.
• Not best everywhere. On NoCap CIDEr, InstructBLIP scores 123.1 against GLaMM's 106.8 tab. 6. The paper's claim of favourable captioning should be read as a claim about SPICE and Flickr30k CIDEr only.
• Human-annotated data in the headline model. GLaMM is trained with 1K manually annotated GCG images; GLaMM† excludes them and scores lower on every GCG metric tab. 3.
• Dataset rights need checking. GranD_f repurposes Flickr30k, RefCOCOg and PSG, with GPT-4 writing the captions §4.5 app. A.3. GranD is built on SA-1B images with Vicuna-v1.5-13B, LLaVA and other open models app. A.4. This card did not check those upstream terms, the Vicuna and LLaVA model licences or the GPT-4 terms, so do not treat the dataset as cleared for commercial use.
• Inference cost. The hidden-state pass runs one batch item at a time GLaMM.py, and the grounding image encoder runs once per image GLaMM.py. Expect latency that grows with the number of images in a batch.
• Mask definitions differ. The paper says masks are binary §3.1; training uses logits GLaMM.py. Threshold at 0 when you evaluate infer.py.

Lineage

• Builds on: GPT4RoI, arXiv:2307.03601, for the region encoder design app. A.2; LISA, arXiv:2308.00692, for the grounding encoder and pixel decoder app. A.2; SAM, arXiv:2304.02643, for the grounding image encoder §3.1; CLIP for the global encoder §3.1; the LLaVA-1.5 projector MLP app. A.2.
• Compared against: LISA, BuboGPT and Kosmos-2 on GCG tab. 3; LISA, CRIS, LAVT, GRES, X-Decoder and SEEM on referring segmentation tab. 4; GRiT and GPT4RoI on region captioning tab. 5; BLIP, BLIP-2, InstructBLIP and others on image captioning tab. 6; LLaVA-1.5 and Qwen-VL on conversation QA tab. 8.
• Superseded by: not known from content on this wiki.