ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

In plain words

Chatbots that look at pictures are good at describing the whole scene, but they often fail when you ask about one spot, such as "the person on the left" or "the object under this arrow". Earlier fixes added extra machinery that encodes coordinates or region features alongside the image.

This paper takes a simpler route. The marks are drawn onto the picture itself: a box, a circle, an arrow, a scribble or a filled mask. The image encoder then sees the marks as ordinary picture content, and no separate region module is needed. The authors observed that a pretrained image encoder already responds to drawn marks, so the design builds on that.

The model is trained on question-and-answer examples in which marks have been drawn on existing region-annotated images, so it learns to read the marks and answer about the marked areas. Some of the training examples were written by a strong multimodal model that was shown the marked images. The authors also made a new test set, with marks drawn by people as well as synthetic boxes. The model does well on several region-reasoning benchmarks. It suits cases where a user points at something, rather than giving coordinates.

TL;DR

• Visual prompts (rectangles, ellipses, points, triangles, masks, mask contours, arrows, scribbles) are alpha-blended into the RGB image, and the composite is fed to a LLaVA-style model §3.1 eq. 1 fig. 2.
• The vision encoder is CLIP ViT-L/14 at 336 px, using features from several layers, with a 2-layer MLP connector; the language model is Vicuna v1.5 at 7B or 13B §3.1 §5.1.
• Training is three stages: connector pre-training on 558k captioned pairs, instruction tuning on LLaVA-1.5 data plus 520k region-prompt examples, and a short fine-tune on 26k GPT-4V-augmented examples §5.1 §3.2 §3.3.
• Author-reported: Visual7W 87.91% and PointQA 71.77% (13B), VCR Q→AR 78.93% (7B base, fine-tuned), ViP-Bench 48.3% (13B, synthesised boxes) §5.2 §6.

Problem

Whole-image instruction models such as LLaVA receive no information about which part of an image a question refers to. Prior region-aware models either give textual coordinates, learn positional embeddings, or use ROI features, and the authors argue these are awkward for users because they accept mostly fixed forms such as boxes §1 §2.

No instruction dataset with arbitrary visual prompts and matching text answers existed when the paper was written, and no benchmark tested region understanding under such prompts §3.2 §4.

Method

Overview. The model does not add a region encoder. The visual prompt is drawn into the image, and the rest of the pipeline is a standard LLaVA-style model §3.1 fig. 2.

1. Overlay the prompt. The prompt layer P_v is alpha-blended onto the image X_v, and only pixels under the prompt change §3.1 eq. 1.
2. Encode with CLIP. The composite goes through the CLIP encoder. The paper keeps one early layer, for geometry, and four deeper layers, for semantics, and concatenates them §3.1.
3. Normalise and project. A LayerNorm and a 2-layer MLP map the concatenated features to visual tokens for the language model §3.1 §5.1.
4. Train autoregressively. The language model is trained to maximise the likelihood of the answer tokens given the composite image and the instruction §3.1 eq. 2.

# Build the visual tokens for one image with its drawn prompts  [§3.1] [eq. 1]
# Input:  image (H, W, 3); prompt_layer (H, W, 4) RGBA; alpha in [0, 1]
# Output: visual tokens (P, d_llm), P patches per image

def visual_tokens(image, prompt_layer, alpha):
    # 1. Blend the prompt over the pixels it covers only  [eq. 1]
    composite = blend(image, prompt_layer, alpha)     # (H, W, 3)
    # 2. Run the frozen CLIP encoder and keep several layers  [§3.1]
    layers = clip_hidden_states(composite)            # per layer (P, d)
    picked = [layers[i] for i in layer_ids]           # layer_ids: see Pitfalls
    # 3. Concatenate along channels and normalise  [§3.1]
    features = layer_norm(concat(picked, axis=-1))    # (P, 5 * d)
    # 4. Two-layer MLP maps to the language model width  [§5.1]
    return mlp(features)                              # (P, d_llm)

The loss is the standard next-token objective over the answer tokens:

P(\mathbf{X}_{\mathrm{a}} \mid \hat{\mathbf{X}}_{\mathrm{v}}, \mathbf{X}_{\text{instruct}}) = \prod_{i=1}^{L} P_{\boldsymbol{\theta}}\bigl(x_i \mid \hat{\mathbf{X}}_{\mathrm{v}}, \mathbf{X}_{\text{instruct}}, \mathbf{X}_{\mathrm{a},<i}\bigr) eq. 2

Here θ denotes the trainable parameters and L the answer length §3.1 eq. 2.

Visual prompts. Eight prompt types are used: rectangle, ellipse, point, scribble, triangle, mask, mask contour and arrow §3.2 §5.1 fig. 3. Attributes (colour, thickness, alpha) are randomised, and the assignment of type and attributes to each region is random during training §5.1.

• Ellipses are enlarged by a random factor between 1 and 1.5 of the box size §3.2 visual_prompt_generator.py.
• A scribble is a Bézier curve whose anchor points are sampled inside the object mask. The paper describes three anchors on a quadratic curve §3.2; the code uses four points on a cubic curve (see Paper vs code).
• Regions with only a bounding box get rectangle, ellipse or arrow prompts; regions with a mask get all eight §3.2. The code's per-dataset subsets differ (see Paper vs code).

Region-level data. The 520k instruction pairs come from RefCOCOg, PointQA (LookTwice), Visual Genome, Flickr30k Entities, VCR and Visual7W, drawn from their training splits §3.2. Optional extra data: 13k region examples written by GPT-4V from two images, the original and one with prompts drawn on it, plus the ground-truth text. Prompts are referred to with placeholders such as "within red mask" and swapped for the real prompt during training §3.3 [app. Effect of Optional GPT-4V Data].

# Region data curation with GPT-4V  [§3.3]
# (see [app. Effect of Optional GPT-4V Region-Level Instruction Data])
# Input:  image with ground-truth region text
# Output: (visual prompt, text prompt, text output) triplet

def curate_triplet(image, region_text):
    # 1. Draw the prompt onto a copy of the image  [§3.2]
    marked = draw_prompt(image, region)
    # 2. Ask GPT-4V with both images, the ground truth and system messages
    #    [§3.3]
    answer = gpt4v(image, marked, region_text, system_messages)
    # 3. Keep the triplet; placeholders are swapped for prompts at training
    #    [§3.3]
    return (prompt_name, text_prompt, answer)

Hyperparameters

Values are from the paper, the appendix, or the official code at code_commit. Where the paper gives nothing, the cell says "not stated". Code values are not the paper's claims unless the paper says so.

The 13B training scripts are not in the repository; only the 7B Vicuna, Llama-3 and Phi-3 scripts are. The 13B optimisation values are therefore not verified.

Results

Author-reported throughout. No variance across seeds is reported, except for the judge check in the appendix.

Visual7W (test set; "Generalist" means not trained on the target dataset) §5.2. Generalist models are marked with a tick.

PointQA-LookTwice (test set; † zero-shot) §5.2.

VCR validation (fine-tuned from ViP-LLaVA-Base-7B; accuracy in %) §5.2.

ViP-Bench (303 image-question pairs; GPT-4 judge; accuracy in %) §6. The "Synth" block uses tight boxes from SAM; the "Human" block uses prompts drawn by people, such as arrows and circles §4 §6. Format: VP = visual prompt; Coor = textual coordinates; ROI = CLIP ROI features; Dis = discrete position tokens.

The benchmark's capability mix (counts; a sample can carry several capabilities, so the counts sum above 303) app. Statistics of ViP-Bench: recognition 240, OCR 89, knowledge 59, math 31, relationship 28, language generation 16.

Ablations and other reported numbers.

• Single CLIP layer (second-last, as in LLaVA) vs multi-layer, Base-7B §5.4: Visual7W 82.44 → 86.04; PointQA 70.23 → 70.86; VCR Q→A 84.64 → 87.66.
• Visual prompt type, 7B model, per the appendix table [app. Performance across Different Visual Prompts]. The table mixes rows that match Base-7B and rows that match ViP-LLaVA-7B (see Pitfalls).

• Overlay vs. separate image tokens app. Effects of Splitting Overlaid Images into Two Separate Image, 7B: overlay 86.04 / 70.86 / 45.5 / 46.8 vs separate 85.98 / 70.67 / 44.7 / 48.0 (Visual7W / PointQA / ViP-Bench@Box / ViP-Bench@Human).
• Visual prompts vs. textual coordinates, 7B [app. Comparing Visual Prompts with Coordinates]: Visual7W 86.04 vs 86.36; PointQA 70.86 vs 61.4; ViP-Bench@Box 45.5 vs 42.6 (prompts, coordinates).
• Input resolution and LLM, same encoder as GPT4ROI [app. Effects of Input Resolution and LLM]: ViP-LLaVA-Base-7B at 336 px with Vicuna v1.5, 86.04 and 45.50; at 224 px with Vicuna v1.1, 81.80 and 42.28; GPT4ROI-7B at 224 px, 81.83 † and 35.14.
• Original image tokens added to the overlay tokens §5.4: VCR QA accuracy 81.63% with both sets vs 82.47% with overlay tokens only (Base-7B).
• Test-time prompt ensembling app. Potential of Visual Prompt Augmentation, Base-7B: Visual7W 86.04 → 87.44; PointQA 70.86 → 71.62.
• GPT-4V region data app. Effect of Optional GPT-4V Region-Level Instruction Data: Base-7B → 7B: Visual7W 86.04 → 86.09; PointQA 70.86 → 71.31; ViP-Bench@Box 45.5 → 48.4; ViP-Bench@Human 46.8 → 48.3. Base-13B → 13B: Visual7W 87.54 → 88.28; PointQA 72.15 → 71.77; ViP-Bench@Box 48.2 → 48.3; ViP-Bench@Human 47.0 → 48.2.
• Arrow direction app. Understanding Arrow Direction: on 3,520 COCO-derived pairs, ViP-LLaVA-13B reaches 90.28%.
• Region captioning on RefCOCOg app. Region Captioning: Base-7B METEOR 16.6 and CIDEr 105.9, vs GLaMM 16.2 and 105.0 and Kosmos-2 14.1 and 62.3.
• Judge stability app. Assessment of GPT-4 as a Judge: five runs of the judge on Base-7B, overall score variance 0.1.
• LLM backbones app. Impact of Different LLMs on Image-level and Region-level Benchmarks: ViP-LLaVA with each backbone, Visual7W / PointQA / ViP-Bench@Box / ViP-Bench@Human:

Overall claim. The paper says ViP-LLaVA is best among open models on the ViP-Bench breakdown except GPT-4V, and state of the art on Visual7W, PointQA and VCR among the compared models §5.2 §6.

Paper vs code

Code: https://github.com/mu-cai/vip-llava at 77079b1e1990472598f1f8a6bb3f7f470100d191. The project page links this repository, and its owner is the first author. The repository's LICENSE is Apache-2.0. It ships the llava package with the visual-prompt generator and the training and evaluation scripts.

• CLIP layer indices. Paper: layer 6 and layers 15, 18, 21, 24 §3.1. Code: hidden-state indices -2, -5, -8, -11 and 6 clip_4layer_encoder.py. The early layer matches. Taking the paper's numbers as 1-based block outputs, the code's deeper layers are 14, 17, 20 and 23, one block earlier. The paper does not state its indexing convention, so this is not certain.
• Alpha range. Paper: alpha in [0.5, 1] §5.1. Code: integers 96–255 for non-mask prompts (about 0.38–1.0) and 48–128 for masks (about 0.19–0.50) visual_prompt_generator.py. Visual7W evaluation uses a fixed alpha of 128 v7w.sh.
• Scribble. Paper: three anchor points on a quadratic Bézier curve §3.2. Code: four random points on a cubic Bézier curve visual_prompt_generator.py.
• Prompt sets per dataset. Paper: bounding-box-only data gets rectangle, ellipse or arrow; mask data gets all eight §3.2. Code: RefCOCOg gets all eight; VCR gets seven (no point); Flickr30k gets rectangle, ellipse and arrow; VG relationships get rectangle and ellipse; Visual7W and PointQA get rectangles only, in a fixed-width "constant" style visual_prompt_organizer.py. Selection is uniform visual_prompt_organizer.py.
• Judge score scale. Paper: scores from 0 to 10 §4. Code: the judge prompt asks for 0.0–1.0 and rejects values outside [0, 1] vip-bench_evaluator.py vip-bench_evaluator.py. Results are means times 100 vip-bench_evaluator.py. A judge answer that still fails after five tries is scored 0 vip-bench_evaluator.py.
• Judge model. Paper: "GPT-4 text model" §4. Code: gpt-4-0613 vip-bench_evaluator.py.
• Decoding. Paper: greedy decoding for open-source models §6. The benchmark script passes temperature 0 vipbench.sh; the loader's default is 0.2 with sampling only when the temperature is above 0 model_vqa_loader_vip.py model_vqa_loader_vip.py.
• Stage 3 learning rate. The paper gives none §5.1. Code: 2e-7 for stage 3, 2e-5 for stage 2 finetune_stage3.sh finetune_stage2.sh.
• Consistent parts. Vision tower frozen clip_4layer_encoder.py; 2-layer MLP with LayerNorm before it builder.py builder.py; ellipse enlargement in [1, 1.5] visual_prompt_generator.py; stage-1 data 558k §5.1 pretrain.sh.

Pitfalls

• Layer indices. The paper's list and the code's indices differ by one for the deeper layers (see Paper vs code). Implement the code's indices if you want the released weights to behave the same, and say so; the paper alone does not fix the convention.
• Alpha. Use the code's ranges if you want to match the released training; the paper's [0.5, 1] is not what the code samples. Visual7W evaluation uses 128.
• Stroke sizes are relative. Line widths and radii scale with the image size against a 336-pixel anchor visual_prompt_generator.py. The paper does not state this, so a resized input changes the mark thickness.
• Blending. Eq. 1 is a per-pixel blend. The code alpha-composites an RGBA layer visual_prompt_generator.py. The two should agree where the prompt is drawn; anti-aliasing and mask fill are not specified in the paper.
• Constant-width prompts for Visual7W and PointQA. Training on those datasets uses rectangles drawn at a fixed width visual_prompt_organizer.py. The paper does not describe this.
• Data count. The stated total is 520k §3.2; the listed components sum to 522k (80k + 37k + 80k + 30k + 213k + 82k). Treat the total as approximate.
• Arrow placement. The paper's constraint that the arrowhead lies in "[(−W/2, −H/2), (W/2, H/2)]" is written in centred coordinates, and the reading is unclear §3.2. The code caps arrow length at 50 px at the 336 anchor visual_prompt_generator.py.
• Prompt-type table mixes models. The caption says it is for the 7B model of ViP-LLaVA, but its Rectangle row equals ViP-LLaVA-Base-7B on Visual7W and PointQA (86.04, 70.86), while its Ellipse row equals ViP-LLaVA-7B on Visual7W (86.6). Check which model each number belongs to before using the table.
• Visual7W numbers differ between tables. The main Visual7W table gives 86.60 (7B) and 87.91 (13B). The GPT-4V-data appendix table gives 86.09 (7B) and 88.28 (13B) for the same models. The main table is the one the abstract refers to.
• VCR model. The VCR table uses ViP-LLaVA-Base-7B, not ViP-LLaVA-7B §5.2.
• Single runs. Apart from the judge check, no seed variance is reported, so small gaps such as the overlay ablation cannot be separated from run-to-run noise.
• Judge and GPT-4V. ViP-Bench scores come from a GPT-4 judge, and the GPT-4V rows are GPT-4V outputs. The judge is a closed model, and a change of version changes the numbers (see Paper vs code).
• Appendix section labels. The supplement's sections are cited here by title, because the compiled source does not expose section letters in the LaTeX.
• Figure and table numbers. Figure and table numbers in this card come from the order in the LaTeX source. Check them against the compiled PDF before citing a number.
• Benchmark source images. ViP-Bench images come from MM-Vet, MMBench and Visual Genome §4. The paper does not discuss their licences for commercial reuse.
• Training data and GPT-4V output. The region data was generated with GPT-4V. The paper does not state the terms that apply to that output. The source datasets (RefCOCOg, VCR, Visual7W, Visual Genome, Flickr30k Entities, PointQA, LLaVA data) each carry their own terms, which are not checked here. Confirm them before commercial reuse of the data or of models trained on it.
• Model and data links. The project page lists the training data and ViP-Bench as Hugging Face datasets (mucai/ViP-LLaVA-Instruct, mucai/ViP-Bench). Their licences were not checked.

Lineage

• Builds on: LLaVA and LLaVA-1.5, whose architecture, instruction-data recipe and training stages this model follows §5.1; CLIP as the vision encoder §3.1; and prior work showing that CLIP attends to drawn circles and rectangles §3.1.
• Related: region encoders that the paper contrasts with its design, namely coordinate text (Shikra), learned positional embeddings (Kosmos-2) and region ROI features (GPT4ROI) §1 §2. The text-marker idea draws on Set-of-Mark app. Generalization to New Attributes.
• Compared against: Shikra, GPT4RoI, Kosmos-2, Qwen-VL, InstructBLIP, LLaVA-1.5, GPT-4V, and specialist models on Visual7W and VCR §5.2 §6.
• Superseded: not known from content in this wiki.