VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

In plain words

Large language models follow instructions written in plain text. This paper asks whether the same approach can drive computer vision tasks such as finding objects, outlining them, or describing a picture. The trick is to make both the input and the answer look like text. The image is turned into a fixed set of learned tokens, shaped by the words of the instruction. The answer is written as a short sequence of word-like tokens: a class token and location tokens for each object. A language model is trained to write that sequence.

Because the task is written as a sentence, changing the sentence changes the job. The same model can be told which classes to look for, how many outline points to use for a mask, or to write a caption instead. Object detection and instance segmentation work this way too. With a ResNet-50 backbone the model reaches 44.6 box AP on COCO, and with a much larger backbone 60.2.

TL;DR

The authors cast vision-centric tasks (detection, instance segmentation, visual grounding, captioning, VQA) as text generation for an LLM decoder §3. A language-guided tokenizer produces M = 100 image tokens conditioned on the instruction §3.3, and Alpaca-7B with LoRA writes class and coordinate tokens in the output format that the instruction names §3.4. Author-reported COCO box AP is 44.6 with ResNet-50 and 60.2 with InternImage-H tab. 1. The repository linked by the paper does not contain this method at the pinned commit, so every detail below comes from the paper alone.

Problem

Vision foundation models are usually adapted to each downstream task, and generalist vision models are still tied to pre-defined task formats §1. Visual prompt methods use visual masks that do not match the form of language instructions §1 §2. API-based systems that chain a language model with vision tools are not end-to-end and are limited in fine-grained detail §2. The authors want one generalist model whose task can be changed with language instructions, as with LLMs §1.

Method

The model has three parts: a unified language instruction, a language-guided image tokenizer, and an LLM-based open-ended task decoder §3.1 fig. 3. The whole pipeline is end-to-end, with no tool calls §3.1.

Unified language instruction §3.2 app. A

• Vision-language tasks use plain prompts. Captioning takes a fixed prompt with the image placeholder. VQA takes a prompt that embeds the question placeholder §3.2.
• Vision-only tasks combine a task description with an output format. Task descriptions were generated from seed instructions with placeholders, by self-instruct, and one description is drawn at random per training example §3.2 app. A.
• Every perception output is a tuple (C, P): a class index C and N points P. The class index is an integer starting at 0. The point coordinates are discretized into integers within [-range, range] §3.2.
• Object detection and visual grounding use N = 2 points, the top-left and bottom-right corners §3.2. Instance segmentation uses N > 8 points along the object boundary as a mask §3.2. Keypoint detection can be cast the same way §3.2.
• Placeholders used in instructions are <class> (the category set), <range>, <image> and <question> §3.2 app. A.

Language-guided image tokenizer §3.3

1. The image goes through a backbone (ResNet is the example given), which gives visual features at four scales §3.3.
2. A text encoder (BERT is the example given) encodes the prompt §3.3.
3. Text features are injected into each scale through cross-attention, which makes the visual features language-aware §3.3.
4. A transformer based on Deformable DETR, with M randomly initialised learned queries, turns the features into M image tokens. Each token has an embedding e_i and a location l_i §3.3 §4.1.

Dropping BERT from the tokenizer lowers RefCOCO P@0.5 from 84.1 to 48.1, so the text path matters for grounding tab. 3a.

LLM-based open-ended task decoder §3.4

• The decoder is Alpaca, an instruction-tuned LLaMA model §3.4 §4.1. The authors list three weaknesses for vision: few digit tokens for locations, several tokens per category name, and a causal structure that suits perception tasks badly §3.4.
• Vocabulary expansion: 1025 location tokens <p-512> to <p512>. Token <p i> is the discrete offset i from the location l_i of an image token, and the relative size is i / 512 of the image height or width §3.4.
• 512 class tokens <c0> to <c511> replace category names. The instruction maps each name in the category set to one class token, for example "person":<c0> §3.4.
• Output-format-as-query: the LLM first parses the output structure from the instruction, for example the query tokens for class and corners in detection, or the beginning-of-string token for captioning. These query tokens are then fed to the decoder, so that the outputs are generated as in parallel decoding rather than token by token §3.4 §4.2 fig. 4.
• LoRA is used for parameter-efficient training §3.4 app. D.
• Because every output is a token, both vision-language and vision-only tasks use cross-entropy, as for language §3.4.

Training §4.1 app. B app. C

The total loss is the sum of the tokenizer loss and the decoder loss eq. 1 app. B.

# Training loss  [app. B] [eq. 1]
# Input:  image tokens, decoder logits, ground-truth objects
# Output: scalar loss, L = tokenizer loss + decoder loss

def training_loss(image_tokens, decoder_logits, objects):
    # Tokenizer: category-agnostic focal loss on the query scores  [app. B]
    # How queries are assigned to objects is not stated [app. B].
    tokenizer_loss = focal_loss(query_scores, query_labels)
    # L1 loss on the centre-point regression of the queries  [app. B]
    tokenizer_loss = tokenizer_loss + l1_loss(query_centres, centre_targets)
    # Set outputs (boxes, masks): bipartite matching aligns the predicted
    # sequences with the ground truth  [§3.4] [app. B]
    matches = bipartite_matching(decoder_logits, objects)
    # Word outputs use plain next-token supervision  [app. B]
    # Set outputs use the matched targets, with the same cross-entropy  [app. B]
    decoder_loss = cross_entropy(decoder_logits, matched_targets(matches))
    return tokenizer_loss + decoder_loss

The two-stage schedule is as follows. Stage 1 initialises the tokenizer from Deformable DETR and BERT weights and the decoder from Alpaca-7B weights. It trains the backbone and tokenizer, freezes most of the LLM, and keeps only the LoRA parameters trainable. Stage 1 focuses on detection, with random task descriptions and random object categories §4.1 app. C. Stage 2 starts from the Stage 1 weights, freezes the visual backbone, and adds detection, instance segmentation, visual grounding, captioning and VQA together §4.1 app. C. Two-stage training converges faster than single-stage training according to the schedule comparison, but no numbers are given fig. 5.

# Inference for detection-style outputs  [§3.1] [§3.4] [§4.2]
# Input:  image tokens, instruction text
# Output: one (class, points) tuple per predicted object

def decode_objects(image_tokens, instruction):
    # The LLM parses the output format named in the instruction  [§3.4]
    format_queries = parse_output_format(instruction)
    # Decode all predictions in parallel from the format queries  [§3.4]
    # [§4.2] reports this as the efficiency advantage.
    raw_outputs = decoder(format_queries, image_tokens)
    # Offsets are relative to a centre point; the mapping from bins to pixels
    # is not fully stated [§3.2] [§3.4]. See Pitfalls.
    return [to_tuples(output) for output in raw_outputs]

Key equation:

\mathcal{L} = \mathcal{L}_{tok} + \mathcal{L}_{dec} eq. 1 app. B

Hyperparameters

Results

All numbers are author-reported. Setup: COCO2017 for detection and instance segmentation §4.1. RefCOCO, RefCOCO+ and RefCOCOg are combined for training, with over 120k referred objects, and evaluated on the RefCOCO validation set §4.1. Captioning is trained on COCO Caption and evaluated on the COCO Karpathy test split §4.1. VQA is shown only qualitatively, because LLaVA-Instruct-150K is not compatible with a standard VQA benchmark §4.1 fig. 10. The generalist rows come from one shared-parameter model whose task is switched by changing the instruction §4.2.

Headline numbers:

• Box AP 44.6 with ResNet-50, which is 1.4 above Pix2Seq at 43.2 with the same backbone §4.2 tab. 1.
• Box AP 60.2 with InternImage-H, close to the detection-specific state of the art §4.2 tab. 1.
• Visual grounding P@0.5 of 80.6 with ResNet-50 and 86.7 with InternImage-H, on RefCOCO validation §4.2 tab. 1.
• Captioning BLEU-4 31.0 and CIDEr 112.5 with ResNet-50; 32.1 and 114.2 with InternImage-H §4.2 tab. 1. No beam search was used §4.2.

Table 1. Standard vision-centric tasks tab. 1. "sep" means trained separately on each task. "Open-ended" is the paper's own column.

Table 2. Object-level and output-format customization tab. 2. Both on VisionLLM-R50, on COCO minival.

(a) Box AP, varying the number of classes in the category set. Asterisked rows report the mean AP over that many classes, not full-COCO AP.

(b) Mask AP, varying the number of boundary points.

Table 3. Ablations in the main text tab. 3. R50 backbone, object detection with random classes and task descriptions, COCO2017 §4.4.

(a) Text encoder in the tokenizer.

(b) Tokenization method.

(c) Number of location bins.

Table 4. More ablations in the appendix tab. 4 app. D. Object detection with R50 on COCO2017 unless noted.

(a) Randomness in training prompts.

(b) LoRA and randomness.

(c) Number of image tokens.

(d) Sequence-to-sequence decoding. The paper's method is the "no" row, as discussed in the text app. D. The "yes" row has no AP reported.

(e) Large vocabulary.

Qualitative results: the four example outputs in the introduction show a custom class set, an editable output format, a controllable caption length and VQA with reasoning fig. 2. Robustness to prompts: eight prompts give AP between 44.7 and 44.8, with a 0.1 AP gap fig. 6. Per-category curves show that classes outside the instruction's category set score close to zero [fig. 7]. Mask refinement with more points fig. 8. Switching the category set changes the detected classes fig. 9. Caption length and VQA examples are in fig. 10. These figures are qualitative only.

Paper vs code

The paper links https://github.com/OpenGVLab/VisionLLM, which has an Apache-2.0 LICENSE at the root. That repository does not contain an implementation of this method at the pinned commit 148cc93. The VisionLLM/ folder holds only a README, and its schedule lists "Release code and models" as unchecked README.md. The README also says that VisionLLM will be integrated into InternGPT, as a TODO README.md. The abstract says the code "shall be released" [abstract]. The root README lists VisionLLM and VisionLLM v2 as a series README.md. The VisionLLMv2/ folder is a different paper's implementation (arXiv:2406.08394), and nothing from it is used on this card.

A search of the clone for the v1 location and class token names, for Alpaca and for "language-guided" found no matching implementation. The only hit was an unrelated class name in an mmdet file. So no hyperparameter or algorithm step on this card has been checked against code. Everything comes from the paper.

Pitfalls

• Headline 60+ mAP uses the large backbone. The "over 60% mAP on COCO" claim is for InternImage-H, which scores 60.2 tab. 1. The ResNet-50 model that most ablations use scores 44.6. Name the variant every time.
• Ablation baselines are not one model. The no-randomness baseline is 45.2 tab. 4a. The separately trained model in Table 1 is 44.8, and the joint model is 44.6. These are different settings, so compare numbers only within one table.
• The location-token range is stated two ways. The default is 1025 tokens, from -512 to 512 §3.4. The ablation text describes 257 bins as -128 to 128 and 2049 as -1024 to 1024 §4.4. Use 1025 tokens with the range -512 to 512, which matches the default. The paper's ablation is the only source of other values.
• The offset reference is ambiguous. Offsets are "relative to the center point" §3.2 app. A. The paper does not say which centre is meant: the object's box centre, or the location l_i of the image token. It also says the bin scale is i / 512 "relative to image height or width" §3.4, so the mapping from bins to pixels is not fully defined. Code would settle this, but none is released.
• The range value for detection is not stated outright. The instructions use <range> §3.2, and the location tokens reach ±512 §3.4. Taking <range> as 512 is an inference from the tokens, not a stated fact.
• Grounding numbers come from different configurations. The table shows 80.6 for the joint ResNet-50 model and 84.4 for the separately trained one tab. 1. The ablation shows 84.1 with an unfrozen BERT tab. 3a, but the paper does not say which configuration that is. Choose one baseline and say which.
• Table 2a's rows are not comparable. The 10-, 20- and 40-class rows are mean AP over subsets of classes, not full-COCO AP tab. 2a. The paper does not explain why the 20-class row (52.7) is higher than the 10-class row (48.9).
• The mask-AP gap is only partly explained. Mask AP75 is low, as the paper notes. It gives three possible causes: integer quantisation, a limit of points, and point-based methods in general §4.2. No experiment isolates them.
• The sequence-to-sequence comparison has no AP. The paper says the seq2seq loss is hard to converge, and reports no AP app. D tab. 4d. The advantage of output-format-as-query therefore rests on qualitative evidence.
• LoRA is needed under random prompts. Without LoRA and with randomness, AP is 1.2 tab. 4b. The paper reads LoRA as a bridge between language and visual tokens, but does not measure that app. D.
• The BERT citation points to another paper. The bibliography entry for "BERT-Base" is Bertasius et al. 2021, on space-time attention for video §4.1 [bib: egbib.bib]. The text in §3.3 cites BERT by Devlin et al. Treat the bibliography key as an error, and use BERT-Base as the text intends.
• Efficiency is claimed but not measured. Parallel decoding is said to make detection more efficient §4.2, but no latency or throughput numbers are given.
• Captioning is not decoded with beam search. Neither beam search nor CIDEr optimisation is used §4.2. Compare with caution against baselines that use them.
• Instance-mask point counts vary. The paper says N > 8 §3.2, uses 8 to 24 points in the experiments §4.2, and shows 14, 20 and 24 in the example instructions app. A. Table 2b covers 8, 14, 16 and 24 points tab. 2b.
• Stage 2's freezing is partial. Only the backbone is said to be frozen §4.1 app. C. Whether any other tokenizer weights are frozen is not stated.
• Prompt robustness is shown for one model and one task. The eight prompts are all detection prompts fig. 6 app. D.
• The LaTeX typo "federal loss" app. D probably means federated loss. The paper does not describe it, so no implementer should try to add it.
• Numbers follow the LaTeX source order. Table and figure numbers were counted from the source (arxiv_clean.tex), not from a compiled PDF. Check them against the PDF before you rely on them.
• Not stated: focal-loss parameters, the bipartite matching cost, LoRA settings, optimizer betas, weight decay, warmup, input resolution, data augmentation, the bin-to-pixel mapping, and the LR schedule's warmup. An implementer will need to choose these, and the choices should be recorded.

Lineage

• Builds on: LLaMA, through the Alpaca-7B decoder [/papers/2302.13971]. Deformable DETR, as the tokenizer's transformer §3.3 (cited as zhu2021deformable, no card). LoRA, for parameter-efficient training §3.4 (cited as hu2021lora, no card). Self-instruct, for task descriptions §3.2.
• Compared against: Pix2Seq, Uni-Perceiver and its variants, UniTab, and Pix2Seq v2 tab. 1. API-based systems such as Visual ChatGPT, MM-REACT, HuggingGPT, InternGPT and VideoChat, which the paper contrasts with its end-to-end design §2 §3.1.
• Follow-up named in the repository: VisionLLM v2 (arXiv:2406.08394), which has no card here README.md.