wikillm

Paper · 2023-05-29 · cs.CV, cs.AI

Contextual Object Detection with Multimodal Large Language Models

ContextDET adds a detection head to a frozen multimodal LLM: each object word the model generates or fills in is turned into conditional queries that a DETR-style decoder uses to output boxes. Introduces the CODE benchmark (cloze, captioning, QA). Author-reported: 48.7 cloze Acc@1 and 10.2 cloze AP@1 on CODE val (OPT-2.7B, ResNet50).

Multimodal language modelObject detectionOpen-vocabulary detectionBipartite matching for set-prediction lossesmultimodalobject-detectionopen-vocabularybenchmark

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A multimodal language model can describe a picture and answer questions about it, but it cannot point at the things it mentions. This paper asks the model to do both. When it talks about a "groom" or a "bride", it should also say where that person is in the image, with a box. The authors call this contextual object detection. The names come from the conversation, so they are not limited to a fixed list of classes.

The method works in two steps: generate, then detect. The language model writes or completes text, and the words that name objects are picked out. For each such word, the model's internal state becomes a query, and a small DETR-style decoder looks at the image and returns boxes for that word. The authors test three tasks: filling in masked object names (cloze), writing a caption, and answering a question.

Most of the system is frozen. The image encoder and the language model keep their pre-trained weights, and the authors train only the lighter parts. In the default setting, only a small share of the parameters is trained. Results are reported by the authors on their own benchmark, so they are best read as evidence that the idea works, not as a ranking against other systems.

TL;DR

  • Adds an end-to-end detection path to a frozen multimodal LLM: the object words it generates or fills in become the queries of a DETR-style decoder that predicts boxes §3.
  • Introduces the CODE benchmark built from Flickr30k and Flickr30k Entities, with masked-name cloze, captioning and QA settings §4.
  • Author-reported: cloze Acc@1 48.7 on CODE val, against 43.2 for the strongest cascade baseline (LLaVA1.5 with GLIP fine-tuned on CODE) tab. 2.
  • The released code is under a non-commercial license and the README says the training scripts are not yet cleaned up. Several hyperparameters and design details differ between paper and code (see Paper vs code).

Problem

Standard detectors such as Mask R-CNN and DETR are trained with closed label sets and cannot name objects outside those sets §1. Open-vocabulary detectors built on CLIP extend the set, but only to a list of novel category names known in advance, not to any word from a human vocabulary §1, §2. Prompting a language model to call external vision experts keeps the models frozen, so their knowledge is not shared end to end §2. The paper also argues that the locate-then-classify design of detectors does not fit contextual use, where the same visual object gets different names in different contexts §1.

The COCO benchmark has 80 classes §4. The CODE benchmark has 10,346 unique object names §4, so a detector trained on fixed classes cannot be compared directly on it.

Method

ContextDET has three parts §3: a visual encoder that makes two kinds of visual tokens §3.1, a pre-trained LLM that reads those tokens together with a task prompt and generates text §3.2, and a visual decoder that turns each object word into boxes §3.3. The text is generated first, and boxes follow from the object words in that text (generate-then-detect).

Visual encoder §3.1. The image passes through a vision backbone, which the paper says is pre-trained and frozen (ResNet, ViT or Swin are named). Two token sets come from the feature map:

  • Local tokens: the feature grid is pooled into p bins by adaptive average pooling and projected to the LLM input size. The result is a fixed number of tokens, one per bin.
  • Full tokens: the feature grid is flattened to m = h × w tokens and encoded by six Transformer layers. These are the keys and values of the visual decoder.

Multimodal context with the LLM §3.2. The LLM input is the local visual tokens followed by the prompt tokens. Prompts differ by task: a sentence with masked object names (cloze), a caption prefix such as "a photo of" (captioning), or a question with an answer slot (QA). The LLM is frozen and decodes autoregressively. The latent it computes for each new position is the state used later eq. 2.

Visual decoder §3.3. Each token the model marks as an object word is given a conditional query set. The paper uses a binary noun classifier to decide which tokens name objects. For such a word with latent e, N learnable object queries q are shifted by a linear projection of e repeated N times. Six cross-attention layers then attend from these conditional queries to the full visual tokens. Two heads output a two-way match score and a box for each query. Only ground-truth boxes whose name is this word enter the matching, so a query set is trained only on the objects its word names. The matching is the DETR bipartite matching, restricted this way.

Training loss §3.4. Four weighted terms: a classification term (binary match vs no match), a box term (L1 or GIoU), a language-modelling term over the vocabulary, and a noun term (is this token an object word). Multi-scale deformable attention and IoU-based label assignment are used to speed up convergence. The vision encoder may start from BLIP-2 weights.

Key equations, numbered as printed in the paper (the section-prefixed numbering is continuous across the paper, so these are (1) to (7)):

p(w_{n+1:L} \mid w_{1:n}) = \prod_{i=n+1}^{L} p_\theta(w_{i+1} \mid w_{1:i}) eq. 1 (see Pitfalls on the index)

e_{n+1} = \mathrm{TransLayers}(w_{1:n}), \quad p(w_{n+1}) = \mathrm{Softmax}(\mathrm{FFN}(e_{n+1})) eq. 2 eq. 3

\bar{q} = q + \mathrm{Linear}(\mathrm{Repeat}(e)) eq. 4

\hat{q} = \mathrm{CrossAttenLayers}(c, \bar{q}) eq. 5

p = \mathrm{FFN}_{\mathrm{cls}}(\hat{q}) \in \mathbb{R}^{N \times 2}, \quad b = \mathrm{FFN}_{\mathrm{box}}(\hat{q}) \in \mathbb{R}^{N \times 4} eq. 6

\mathcal{L} = \lambda_{\mathrm{cls}} \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{box}} \mathcal{L}_{\mathrm{box}} + \lambda_{\mathrm{lm}} \mathcal{L}_{\mathrm{lm}} + \lambda_{\mathrm{noun}} \mathcal{L}_{\mathrm{noun}} eq. 7

The forward pass for one image and prompt, in the paper's terms:

Pseudocode
# Forward pass for one image and one prompt  [§3.1] [§3.2]
# Input:  image (3, H, W); prompt tokens
# Output: a word-level prediction set per object word  [§3.3]
def contextdet_forward(image, prompt_tokens):
    # Frozen backbone gives a feature map  [§3.1]
    feature_map = backbone(image)                   # (d, h, w)
    # Local tokens: p pooling bins, then a linear layer  [§3.1]
    local_tokens = linear_1(adaptive_pool(feature_map, p))   # (p, d1)
    # Full tokens: flatten to m = h * w, then six Transformer layers  [§3.1]
    full_tokens = six_layers(flatten(feature_map))           # (m, d2)
    # Frozen LLM reads [local tokens, prompt tokens]  [§3.2] [eq. 2]
    prefix = concat(local_tokens, embed(prompt_tokens))
    latents = llm_transformer(prefix)
    # Each object word has its latent e  [§3.3]
    for word, e in object_word_latents(latents):
        # Conditional queries: learned queries plus projected e  [§3.3] [eq. 4]
        queries = learned_queries + linear_2(repeat(e, N))   # (N, d)
        # Six cross-attention layers; keys and values from full tokens  [§3.3] [eq. 5]
        decoded = cross_attention(full_tokens, queries)
        # Match score and box per query  [§3.3] [eq. 6]
        scores = head_cls(decoded)                 # (N, 2)
        boxes = head_box(decoded)                  # (N, 4)
    return predictions

Training matches predictions to targets for each object word, then sums the weighted terms:

Pseudocode
# Training loss for one image  [§3.3] [§3.4] [eq. 7]
# Conditional matching: only ground-truth boxes of this word are used
def training_loss(predictions, gt_boxes, gt_words):
    total = 0.0
    for word in object_words:
        # Optimal bipartite matching, restricted to this word  [§3.3]
        targets = select_boxes(gt_boxes, gt_words, word)
        matched = hungarian(predictions[word], targets)
        # Weights: cls 1, box 5, language modelling 1, noun 1  [§3.4]
        total += 1.0 * loss_cls(matched)
        total += 5.0 * loss_box(matched)   # L1 or GIoU  [§3.4]
        total += 1.0 * loss_lm(prompt_tokens)
        total += 1.0 * loss_noun(object_word_flags)
    return total

At inference the model generates until the end-of-sentence token §3.2 eq. 3 and then runs the decoder for each word kept as an object word §3.3. The paper does not state a score threshold for keeping object words at inference.

Hyperparameters

NameValueWhereNotes
Vision backboneResNet50 (default); Swin-B also reported§5 tab. 3frozen per §3.1; code differs, see Paper vs code
Language modelOPT-2.7B (default); OPT-6.7B also reported§5 tab. 3frozen (Fig. 2 caption)
Local bins p9 (default); 4 and 16 tested§5.2 tab. 5grid shape for p = 9 not stated
Full-token encoder layers6 Transformer layers§3.1
Visual decoder cross-attention layers6§3.3
Object queries Nnot stated§3.3
Loss weights λcls, λbox, λlm, λnoun1, 5, 1, 1§3.4 eq. 7
Box lossL1 or GIoU§3.4which one is used for the main results is not stated
OptimizerAdamW§5
Learning rate1e-4§5warm-up and schedule not stated
Weight decay0.05§5
Batch size8§5per GPU or total not stated
Epochs6§5
Hardware4 × NVIDIA A100, one machine§5
Augmentationhorizontal flip with p = 0.5; large-scale jittering§5
Convergence aidsmulti-scale deformable attention; IoU-based label assignment§3.4no values given
BLIP-2 initialisationoptional for the vision encoder§3.4
Matching costsnot stated§3.3
AP IoU thresholds0.50 to 0.95, step 0.05§4COCO convention
AP@5 rulea prediction counts if the ground-truth name is among its top 5 names§4AP@1 uses the top-1 name
CODE splitstrain 665,161 boxes in 29,781 images; val 22,061 in 1,000; test 21,641 in 999§410,346 unique object names
OV-COCO split65 categories: 48 base, 17 novel§5.3
Open-vocabulary queryone yes/no question per class asking whether that class appears in the picture§5.3a class is used when the answer is yes

Results

Author-reported throughout. Setup: CODE val set (cloze, captioning and QA), OPT-2.7B with ResNet50 unless stated §5.1. Baselines are cascades: a captioner (BLIP-2, LLaVA or LLaVA1.5) followed by GLIP, and GLIP-2 for QA and captioning, which cannot run the cloze setting §5.1.

Main CODE result tab. 2. Acc is classification accuracy in percent for cloze; AP is detection average precision in percent.

#MethodCloze Acc@1Cloze Acc@5Cloze AP@1Cloze AP@5QA AP@1QA AP@5Cap AP@1Cap AP@5
1BLIP-2 + GLIP40.469.09.019.15.513.45.27.0
2LLaVA + GLIP41.870.48.618.85.813.74.47.7
3LLaVA1.5 + GLIP42.971.19.219.06.514.05.57.6
4LLaVA1.5 + GLIP (fine-tuned on CODE)43.271.59.119.46.313.95.88.0
5GLIP-2not runnot runnot runnot run2.411.95.76.6
6ContextDET48.773.810.220.58.118.26.49.2

Backbone and LLM variants tab. 3. Parameters in millions; times in seconds per iteration.

#LLMVisionCloze Acc@1Cloze Acc@5Cloze AP@1Cloze AP@5QA AP@1QA AP@5Cap AP@1Cap AP@5Total paramsTrainableT_trainT_test
1OPT-2.7BResNet5048.773.810.220.58.118.26.49.228351830.4370.224
2OPT-2.7BSwin-B54.378.113.125.38.620.08.212.428932410.6250.241
3OPT-6.7BResNet5049.274.511.123.48.07.511.621.369222630.4480.248
4OPT-6.7BSwin-B54.878.613.726.69.023.49.214.069793200.6520.251

Local-token ablations on the CODE val set (note: these rows differ from tab. 2 and 3, see Pitfalls).

SettingCloze Acc@1Cloze Acc@5Cloze AP@1Cloze AP@5Source
Without local tokens30.957.14.013.6tab. 4
With local tokens, p = 948.773.910.421.6tab. 4, tab. 5
p = 448.473.210.120.1tab. 5
p = 1647.572.99.919.4tab. 5

Open-vocabulary detection on OV-COCO tab. 6. AP at IoU 0.5 for novel, base and all classes.

MethodNovelBaseAll
ViLD27.659.551.2
OV-DETR29.461.052.7
BARON34.060.453.5
ContextDET36.865.157.7

The text gives gains over BARON of 2.8, 4.7 and 4.2 points on novel, base and all §5.3, which match the table.

Referring image segmentation, mIoU tab. 7. RefCOCO, RefCOCO+ and RefCOCOg (val, testA, testB for RefCOCO and RefCOCO+; val and test for RefCOCOg).

MethodRefCOCO valtestAtestBRefCOCO+ valtestAtestBRefCOCOg valtest
RefTR74.3476.7770.8766.7570.5859.4066.6367.39
LAVT74.4676.8970.9465.8170.9759.2363.3463.62
PolyFormer75.9677.0973.2270.6574.5164.6469.3669.88
ContextDET76.4077.3974.1671.6775.1465.5269.8970.33

Standard COCO detection tab. 8. Reported as a side result; the paper says it does not match specialised detectors overall.

ModelAPAP50AP75APsAPmAPl
DETR42.062.444.220.545.861.1
Deformable DETR46.265.250.028.849.261.7
ContextDET43.462.947.426.046.956.8

Qualitative results and comparisons with GPT-4, BLIP-2 and LLaVA are in figures only fig. 6 to fig. 10; they are not quantitative.

Paper vs code

The official repository is yuhangzang/ContextDET (commit 9abef13b250af16d0ee0a068e413ae488154ad7d), by the first author README.md. Its license is the S-Lab License 1.0, which allows non-commercial use only LICENSE. This card cites the code only in prose and pseudocode; no code is quoted. The README lists the training scripts as "waiting to be cleaned up" README.md, so the training path may differ from the paper's.

  1. Two model variants exist. The OV-DETR variant uses a BERT masked LM and pools visual tokens to a 2 × 2 grid, giving 4 tokens, with the default vis_len of 4 ov_detr.py main.py. The BLIP-2 variant (OVBLIP2DETR) takes BLIP-2's own visual pathway instead of pooled ResNet bins blip2_decoder.py. The demo and the OV-COCO and RefCOCOg scripts use the BLIP-2 variant app_util.py train_ovcoco_ovblip2.sh. Both variants are named in the repository, but the paper describes only one visual-token design: p pooled ResNet bins with p = 9 as default tab. 5. The BLIP-2 variant has no such pooling, and the BERT variant uses 4 tokens. The OPT-2.7B setting in the paper corresponds to the BLIP-2 variant, since both use OPT-2.7B.

  2. Vision backbone is not frozen. The paper says the backbone is frozen §3.1. The code freezes only the stem and first stage and trains layers 2 to 4 when lr_backbone is non-zero backbone.py. The default lr_backbone is 2e-5 main.py, and train_backbone is true for any non-zero value backbone.py.

  3. Language-model states are detached. The code detaches the LLM hidden states before the projection ov_blip_detr.py. The LLM is frozen anyway, so this matches the paper's frozen LLM. Gradients still never reach it, so the trainable path is the projections and decoder only, which narrows what "end-to-end differentiable" [abstract] covers.

  4. Object-word selection differs. The paper uses a binary noun classifier and a noun loss §3.2 §3.3 §3.4. The code predicts span start and end positions per token with sigmoid heads. At inference a token is a start or end when its probability is above 0.3, with an argmax fallback if none passes ov_blip_detr.py ov_blip_detr.py. The "instance" heads are reached only through an else branch that elif True makes unreachable ov_blip_detr.py ov_blip_detr.py. I infer the start/end heads play the role of the paper's noun term; the paper does not say so.

  5. Training picks one span per image. During training the code samples k = min(k, 1), so one ground-truth span per image is used at each step ov_blip_detr.py. The paper says nothing about which words are sampled.

  6. Conditioning vector. The paper conditions on the single latent e of the word eq. 4. The code averages hidden states over the whole span of a multi-token word ov_blip_detr.py ov_blip_detr.py. In inference it processes at most four spans per decoder pass ov_blip_detr.py.

  7. Where the conditioning enters. The paper adds a projection of e to the learned queries eq. 4. The code adds it to the content embedding after a two-stage proposal path transformer.py. The non-two-stage branch raises NotImplementedError transformer.py. The paper does not describe two-stage proposals, and the scripts pass --two_stage improved_baseline.sh.

  8. Loss terms. The weights in the code are: classification from cls_loss_coef (default 2; 1.0 in the improved baseline; 2.0 in the OV-COCO and RefCOCOg scripts) main.py improved_baseline.sh train_ovcoco_ovblip2.sh; box L1 at weight 5 main.py; GIoU at weight 2 main.py; start and end terms at weight 1 ov_blip_detr.py. The paper gives λcls = 1 and says the box term is L1 or GIoU §3.4; the code uses both. The paper's classification is a binary softmax; the code uses sigmoid focal loss with alpha 0.25 and gamma 2 set_criterion.py.

  9. Language-modelling loss is not used by the BLIP-2 variant. The BLIP-2 model's loss list is labels, boxes, cardinality and start/end, with no masked-LM term ov_blip_detr.py. The masked-LM loss appears only in the BERT variant ov_detr.py set_criterion.py. That loss maps token 0 to the ignore index set_criterion.py.

  10. Matching costs. The paper gives no matching costs. The code uses class 2, box L1 5 and GIoU 2 main.py. The conditional matcher restricts ground truth to spans equal to the selected span, as the paper describes matcher.py. For the second stage the criterion uses a conditional assigner instead of the matcher set_criterion.py. That assigner uses an IoU threshold of 0.6 and up to k = 4 positives per ground-truth box assigner.py.

  11. Post-processing. The paper describes none for the main results. When the second-stage assigner is on, the code applies non-maximum suppression at IoU 0.7 (batched over labels) and keeps at most 20 boxes per query set ov_blip_detr.py post_process.py post_process.py.

  12. Optimizer and schedule. The code uses AdamW with default learning rate 2e-4 and weight decay 1e-4 main.py main.py main.py. The paper gives 1e-4 and 0.05 §5. The default batch size is 2 main.py, while the paper uses 8. The OV-COCO script uses batch 4, 6 epochs and a step decay at epoch 5 without overriding the learning rate train_ovcoco_ovblip2.sh train_ovcoco_ovblip2.sh. The step decay uses the default factor main.py.

  13. Query count and length. The paper does not state N. The OV-COCO and RefCOCOg scripts use 300 queries train_ovcoco_ovblip2.sh; the improved baseline uses 900 queries and 50 epochs improved_baseline.sh.

  14. Augmentation. Random horizontal flip with p = 0.5 is in the code transforms.py. The paper's large-scale jittering was not located in the dataset transforms by search, so I could not confirm it.

  15. Evaluation code. The CODE evaluation toolkit is in evaluation/, and the annotations are on an external Google Drive folder README.md. I did not read the evaluation sources, so I cannot confirm how the "same meaning" check is implemented.

Pitfalls

  • Eq. (1) index. The left side is p(w_{n+1:L} | w_{1:n}), but the product runs over i = n+1 to L of p(w_{i+1} | w_{1:i}), which conditions on the wrong positions. Implement the standard autoregressive factorisation, not the printed indices.
  • Object-word rule is underspecified. The paper says a binary noun classifier picks object tokens §3.3 but gives no architecture or threshold. The code uses start and end heads with a 0.3 threshold and an argmax fallback (Paper vs code, item 4).
  • Multi-token words. The paper's conditioning uses one latent e per word, and it does not say how a multi-token name is pooled. The code averages over the span (Paper vs code, item 6).
  • Training sampling. The paper says nothing about which object words enter the loss per image. The code uses one random span per image per step (Paper vs code, item 5). Results may depend on this choice.
  • Inconsistent numbers for the same default model. The default configuration (p = 9, OPT-2.7B, ResNet50) is 10.2 AP@1 and 20.5 AP@5 for cloze in tab. 2 and tab. 3, but 10.4 and 21.6 in tab. 4 and tab. 5. The Acc@5 also differs (73.8 vs 73.9). Use one set and state which.
  • Suspicious AP@5 value. AP@5 counts a prediction as correct when the ground truth is among the top five names, so it should be at least AP@1 §4. Table 3, row 3 gives QA AP@1 8.0 and AP@5 7.5. This looks like a typo; check the source before reusing it.
  • Text does not match Table 7. The text claims gains of 0.63 and 0.45 over PolyFormer on validation and test §5.4. The table gives +0.53 on RefCOCOg val, +0.44 on RefCOCO val and +1.02 on RefCOCO+ val; only the test gain (+0.45, RefCOCOg) matches. The paper does not say which split the 0.63 refers to.
  • Small-object claim. Section 5.5 says ContextDET is strong on small objects, but its APs of 26.0 is below Deformable DETR's 28.8 (only DETR's 20.5 is beaten) tab. 8.
  • Same-meaning judgement. The cloze evaluation counts a prediction as correct when it "shares the same meaning" as the ground truth §4, but the paper does not say how that is decided. Accuracy and AP depend on it.
  • Frozen backbone in the paper, trainable in the code. Reproducing the paper's numbers from the released code needs the backbone frozen, which the default script does not do (Paper vs code, item 2).
  • Cost. Training uses four A100 GPUs and six epochs §5. The reported time per iteration for the default setting is 0.437 s train and 0.224 s test tab. 3. Larger LLMs or Swin backbones raise both.
  • Limited training data. The authors say their annotation cost kept training data smaller than in earlier MLLM papers, which may limit results §6.
  • Data availability. The paper calls the datasets public, but the CODE annotations are on an external Google Drive folder README.md. Check the link before building a pipeline.

Lineage

  • Builds on: BLIP-2 for the pre-trained vision encoder and the OPT-based LLM variant §3.4 §5 blip2_decoder.py; DETR for the set-prediction matching and decoder §3.3.
  • Builds on, by name only: OV-DETR (open-vocabulary DETR) and Deformable DETR, multi-scale deformable attention, and DETA, acknowledged in the code README README.md.
  • Compared against: BLIP-2, LLaVA, LLaVA1.5 with GLIP and GLIP-2 cascades §5.1; ViLD, OV-DETR and BARON on OV-COCO §5.3; RefTR, LAVT and PolyFormer on referring segmentation §5.4; DETR and Deformable DETR on COCO §5.5; GPT-4 qualitatively §5.6. The LLaVA baseline is in /papers/2304.08485.
  • Superseded by: not known from the content of this wiki.

Source and license

Adapted from “Contextual Object Detection with Multimodal Large Language Models” by Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, Chen Change Loy (arXiv:2305.18279), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yuhangzang/ContextDET at commit 9abef13b250a (S-Lab License 1.0 (non-commercial)). Any quoted code is shown under that license.