--- kind: concept slug: visual-prompting title: Visual prompting summary: >- Telling a vision model what matters in an input, usually without a new task head. Draw marks onto the image (ViP-LLaVA, VLMnav), tile worked examples into a grid (MAE-VQGAN), stack a solved pair (Painter), learn one frame added to every image for a frozen model (Bahng et al.), or learn prompt features or tokens inside the network (BAT, OVSeg). tags: [multimodal, vision-language, prompting, tracking, segmentation, adapter, in-context-learning, frozen-backbone] papers: ["2203.17274", "2209.00647", "2210.04150", "2212.02499", "2312.00784", "2312.10611", "2411.05755"] related: [multimodal-language-model, visual-instruction-tuning, parameter-efficient-fine-tuning, vision-transformer, visual-object-tracking, open-vocabulary-segmentation, in-context-learning, masked-image-modeling, multi-task-learning, zero-shot-learning, soft-prompt-tuning, vision-language-pretraining, linear-probing] status: draft updated_at: 2026-11-01 --- ## Overview The term covers five mechanisms, and the cards use it differently. - **Drawn marks.** The prompt is a mark such as a box, ellipse, arrow, scribble or mask, drawn onto the image pixels before the image is encoded [arxiv:2312.00784 §3.1]. The encoder sees the mark as ordinary picture content. The mark can point at a region the user chose, or label a candidate action the system generated [arxiv:2411.05755 §3.3]. - **Example grids.** The prompt is a picture of worked examples. An example input, its output and a query are tiled into one grid, and a masked image model fills the empty cell with the answer [arxiv:2209.00647 §3.2]. A person supplies the examples, and no weights change [arxiv:2209.00647 §1]. Painter uses the same idea with a model trained on this format: a solved pair is stacked above the query, and the query's output half is painted [arxiv:2212.02499 §3.3]. - **Learned prompt features.** The prompt is a feature vector computed inside the network. A small adapter sends features from one stream into the other at each layer [arxiv:2312.10611 §3.2]. A user never draws anything. - **Learned prompt tokens.** A binary mask marks the image patches that are pure background. Those patches are replaced by trainable tokens, and the tokens are the only prompt [arxiv:2210.04150 §3.3]. The mask comes from the segmenter's proposals, not from a person. - **Learned pixel overlay.** The prompt is one tensor the size of the input. It is learned once per dataset from labelled examples, then added to every image, while the model stays frozen [arxiv:2203.17274 §3.2.1]. The paper says it need not be readable by people [arxiv:2203.17274 §3.2]. Most of these steer a visual model that already exists. Painter is the exception: it trains one generalist model on the pair format, so the prompt format is part of training [arxiv:2212.02499 §3.2]. The mechanisms differ in who or what creates the prompt and where it enters the network. Whether a person must supply the prompt at inference time is the main constraint. ViP-LLaVA needs the user to point, and MAE-VQGAN needs example pairs from the user. VLMnav and OVSeg derive their marks or masks from the system, and BAT's prompts come from the other stream. Painter's prompt is a pair from its training set, chosen at random or by search, or a learned prompt [arxiv:2212.02499 §3.3]. The pixel overlay is trained once from labelled data and needs no input from a user or the system at inference [arxiv:2203.17274 §3.2.1]. ## Variants ### Drawn marks on the image [ViP-LLaVA](/papers/2312.00784) draws one of eight mark types onto the image: rectangle, ellipse, point, scribble, triangle, mask, mask contour and arrow. The mark is alpha-blended in, and only the pixels under it change [arxiv:2312.00784 §3.1] [arxiv:2312.00784 eq. 1]. Colour, thickness and alpha are randomised in training [arxiv:2312.00784 §5.1]. The composite image goes through a LLaVA-style model with no region module [arxiv:2312.00784 §3.1]. Training uses 520k region-prompt instruction pairs built on existing region-annotated datasets, among them RefCOCOg, PointQA, Visual Genome, Flickr30k Entities, VCR and Visual7W [arxiv:2312.00784 §3.2]. An optional 13k examples were written with GPT-4V from marked images [arxiv:2312.00784 §3.3]. The authors report 87.91% on Visual7W and 71.77% on PointQA-LookTwice for the 13B model [arxiv:2312.00784 §5.2]. [VLMnav](/papers/2411.05755) uses drawn marks for a different job: choosing a direction. A depth-based check proposes collision-free directions, each is drawn as a numbered red arrow on the RGB image, and one VLM call picks a number [arxiv:2411.05755 §3.1] [arxiv:2411.05755 §3.2] [arxiv:2411.05755 §3.3] [arxiv:2411.05755 §3.4]. The VLM is used without fine-tuning [arxiv:2411.05755 §3]. On ObjectNav the authors report 50.4% success with the navigability proposals, 33.2% with static arrows and 29.8% with no arrows [arxiv:2411.05755 tab. 1]. Most of the gain over no arrows therefore comes from choosing the arrows well, not from drawing them. This is one benchmark, with a simulator that allows sliding along walls; without sliding the authors report 12.9% [arxiv:2411.05755 tab. 1]. ### Example grids for an inpainting model [MAE-VQGAN](/papers/2209.00647) is a masked autoencoder whose decoder predicts VQGAN codes. The example input and output and the query go into a 2x2 grid, the bottom-right cell is hidden, and the model predicts its codes [arxiv:2209.00647 §3.2] [arxiv:2209.00647 §4.2]. The model is pretrained on 88,645 arXiv computer-vision figures [arxiv:2209.00647 §3.3]. With one example pair, the authors report 27.17 mean mIoU on Pascal-5i foreground segmentation, against 14.91 for copying the example output [arxiv:2209.00647 §6]. Pretraining matters. With ImageNet alone, the same model reaches 2.22 to 7.07 on the four segmentation splits, against 24.25 to 30.44 with the figures [arxiv:2209.00647 tab. 1]. In the few-shot table, the one-example grid reaches 32.5 on split 0, against 51.3 for FWB, which uses the full labelled base set and a segmentation-specific architecture [arxiv:2209.00647 tab. 3]. ### Stitched pair prompts for a generalist model [Painter](/papers/2212.02499) writes the output of each dense task as an RGB image. Depth, segmentation, keypoints and restoration outputs all become pictures [arxiv:2212.02499 §3.1]. It then trains one ViT-L model to paint a masked output half, given two stacked input-output pairs from the same task [arxiv:2212.02499 §3.2]. At test time, a solved pair is stacked above the query, and the query's output half is painted [arxiv:2212.02499 §3.3]. No task head is added, and a new task needs only new pairs in training [arxiv:2212.02499 §3.1]. Results are author-reported. Painter beats the best listed specialist on NYUv2 depth (RMSE 0.288 against 0.330). It trails the specialist on ADE-20K segmentation (49.9 mIoU against 57.7), COCO panoptic (PQ 43.4 against 57.8) and COCO keypoints (AP 72.1 against 77.2) [arxiv:2212.02499 tab. 1]. Training on all tasks together does better than per-task training on most metrics, with keypoint AP and SIDD SSIM as the exceptions [arxiv:2212.02499 tab. 2]. For learned prompts, the model stays frozen and only the prompt tensors are trained [arxiv:2212.02499 §3.3]. On ADE-20K, random, searched and learned prompts give 49.3, 49.9 and 49.9 mIoU. The paper reports no variance, so that gap may be noise [arxiv:2212.02499 tab. 3]. ### Learned pixel overlay for a frozen model [Exploring Visual Prompts](/papers/2203.17274) learns one additive prompt the size of the input and adds it to every image. The model is frozen, and the prompt is trained by minimising the cross-entropy of the correct label [arxiv:2203.17274 §3.2.1] [arxiv:2203.17274 eq. 1]. Three templates are compared: a pixel patch at a random location, a patch at a fixed location, and a padding frame. The padding frame of width 30 on 224×224 inputs is the default [arxiv:2203.17274 §3.2.2] [arxiv:2203.17274 §6.2]. Training uses SGD at learning rate 40 for 1,000 epochs [arxiv:2203.17274 §3.4]. For CLIP, class names go into a fixed text template, and only the image prompt is optimised [arxiv:2203.17274 §3.3]. On 12 datasets the authors report 78.2 for the prompt, against 54.2 for the text prompt alone and 83.1 for a linear probe [arxiv:2203.17274 tab. 1]. On CLIP over three WILDS sets the prompt averages 58.1, against 62.6 for the linear probe, but it is ahead on Camelyon17, 89.8 against 84.9 [arxiv:2203.17274 tab. 2]. For the Instagram, BiT-M and RN50 vision models, which have no text head, the prompt reaches 25.7 to 32.6 on the same 12 datasets, against 71.7 to 72.6 for linear probes [arxiv:2203.17274 tab. 4]. The paper gives no adversarial bound on the perturbation, so these results say nothing about robustness [arxiv:2203.17274 §3.2.1]. The card's Paper vs code section records that the released code loads only CIFAR-100, so the other datasets cannot be reproduced from it. The same section notes that the released script picks its checkpoint on that test split ([Exploring Visual Prompts](/papers/2203.17274)). ### Learned prompt features across two streams [BAT](/papers/2312.10611) tracks an object from an RGB camera and a thermal camera. It builds on a frozen OSTrack ViT-B/16 [arxiv:2312.10611 §4.1.2]. In every layer, a bi-directional adapter adds to each stream a prompt computed from the other stream, once in the attention stage and once in the MLP stage [arxiv:2312.10611 §3.2] [arxiv:2312.10611 eq. 5]. Only the adapters are trained, 0.32M parameters in total [arxiv:2312.10611 §1]. The authors report PR 70.2 and SR 56.3 on LasHeR, and MPR 86.8 and MSR 64.1 on RGBT234 [arxiv:2312.10611 tab. 1]. The earlier prompt-based tracker ViPT reports 65.1 and 52.5 on LasHeR and 83.5 and 61.7 on RGBT234 in the same table [arxiv:2312.10611 tab. 1]. ### Learned prompt tokens in masked patches [OVSeg](/papers/2210.04150) uses mask prompt tuning in a two-stage open-vocabulary segmenter. Each proposal is cropped, its background pixels are blanked, and every patch that is entirely background is replaced by a learned token. Partly covered patches keep their image tokens, because their boundary pixels still matter [arxiv:2210.04150 §3.3]. The prompts enter at the input and at deeper layers. The default prompt depth is 3 [arxiv:2210.04150 §4.1] [arxiv:2210.04150 app. A.5]. The prompts can be trained with CLIP frozen, starting from the original weights, or on top of a CLIP that has first been fine-tuned [arxiv:2210.04150 §4.1]. On ADE20K-150 the author-reported mIoU is 21.8 for the baseline, 26.5 with prompts only, 28.8 with full fine-tuning only and 29.6 with both [arxiv:2210.04150 tab. 3]. Against deep visual prompt tuning, the appendix reports 26.5 against 25.5 on the same dataset, with the VPT variant using more tokens and about 40% more compute [arxiv:2210.04150 tab. 7]. ## When to use which | Situation | Use | Why | |---|---|---| | Users point at regions in a chat model, and you do not want a region encoder or coordinate text | Drawn marks | The mark sits in the picture, so the existing encoder reads it [arxiv:2312.00784 §3.1]. Cost: region instruction data and a three-stage LLaVA-style training run [arxiv:2312.00784 §5.1] | | Several dense outputs (depth, masks, keypoints, restoration) from one model, with training pairs for each task and a large training budget | Stitched pair prompt | One model with no task head, and a new task needs only new training pairs [arxiv:2212.02499 §3.1]. Training jointly is better than per-task training on most metrics [arxiv:2212.02499 tab. 2]. Cost: a ViT-L, with stitched samples of twice one image's patch count [arxiv:2212.02499 §4.1]. It trails specialists on ADE-20K, COCO panoptic and keypoints [arxiv:2212.02499 tab. 1] | | Your inputs must keep every pixel unchanged | Drawn marks, with care | The overlay alters the covered pixels [arxiv:2312.00784 §3.1]. The authors' separate-token variant scored close, e.g. 85.98 against 86.04 on Visual7W [arxiv:2312.00784 app. Effects of Splitting Overlaid Images into Two Separate Image] | | An agent must pick one of several options that can be read off the image, and you have no training budget | Drawn marks as choice labels | A zero-shot VLM picks a numbered option each step [arxiv:2411.05755 §3.4]. Cost: two VLM calls per step [arxiv:2411.05755 §3.5], and the depth and pose inputs of a simulator [arxiv:2411.05755 §4] | | You have one or a few input-output examples for a simple image-to-image task, no labels for it, and a masked image model pretrained on similar images | Example grid | No weight updates [arxiv:2209.00647 §1]. Results depend on pretraining data: ImageNet alone is far weaker [arxiv:2209.00647 tab. 1]. Labelled specialist baselines still score higher, e.g. 51.3 against 32.5 on split 0 [arxiv:2209.00647 tab. 3] | | A frozen CLIP-style model reachable only through its input, one labelled dataset, and no need to change the model | Learned pixel overlay | One prompt per dataset, trained on the frozen model's label scores [arxiv:2203.17274 §3.2.1]. On CLIP it trails a linear probe on 12 datasets, 78.2 against 83.1 [arxiv:2203.17274 tab. 1], but it leads on Camelyon17 [arxiv:2203.17274 tab. 2]. Cost: 1,000 epochs with gradients through the frozen model; compute not stated [arxiv:2203.17274 §3.4] | | A vision model with no text head, where a linear probe is an option | Linear probe | The overlay reaches 25.7 to 32.6 on 12 datasets, against 71.7 to 72.6 for linear probes on the same models [arxiv:2203.17274 tab. 4]. The paper's own results favour the probe there | | A frozen pre-trained ViT, two input streams, little labelled data, a tight parameter budget | Learned adapter | Only the adapters train, 0.32M parameters, and the RGB-thermal data is small [arxiv:2312.10611 §1] | | The useful stream changes with conditions and you do not want one stream fixed as dominant | Learned adapter | The adapter runs both ways in every layer [arxiv:2312.10611 §3.2]. The gain over a one-way variant is small: BAT-RGB reaches 69.0 PR and 55.4 SR on LasHeR, against 70.2 and 56.3 for BAT [arxiv:2312.10611 tab. 2], so test on your own data | | A pre-trained CLIP must classify masked regions, the mask comes from a proposal model, and you want the CLIP weights kept frozen | Learned prompt tokens | Prompts can train with CLIP frozen [arxiv:2210.04150 §4.1]. The gain is smaller than full fine-tuning, 26.5 against 28.8 on ADE20K-150, so fine-tuning then prompting is the stronger choice if you can afford it [arxiv:2210.04150 tab. 3] | | Anything beyond RGB-thermal tracking | Not covered | The authors validate only on RGB and thermal tracking [arxiv:2312.10611 §5] | ## Implementation sketch The variants share no code. The first block shows how a mark enters the image; the second shows the MAE-VQGAN example grid; the third shows Painter's stitched pair; the fourth shows the pixel overlay; the fifth shows the adapter layer; the sixth shows the prompt tokens. Paper and released code differ on several details, such as the alpha range, CLIP layer indices, adapter normalisation, grid padding, mask ratio, prompt learning rates, where the overlay is added relative to normalisation, and the class-index mapping for models without text. Check each card's "Paper vs code" section before copying any sketch. ```pseudocode # Draw one visual prompt onto the image [arxiv:2312.00784 §3.1] [eq. 1] # Input: image (H, W, 3); mark rasterised as RGBA (H, W, 4); alpha in [0, 1] # Output: composite image (H, W, 3), fed to the usual vision encoder def overlay_prompt(image, mark, alpha): # 1. Rasterise the mark: box, ellipse, arrow, scribble or mask [§3.2] layer = rasterise(mark) # (H, W, 4) # 2. Blend only the pixels the mark covers [eq. 1] return blend(image, layer, alpha) # (H, W, 3) ``` ```pseudocode # Visual prompt as an example grid, read from a masked image model # [arxiv:2209.00647 §3.2] [§4.2] # Input: example pair (x_in, x_out) and query x_q, each a 111 x 111 tile # [arxiv:2209.00647 tab. 3] # Output: predicted output tile for x_q, (111, 111, 3) def grid_prompt(x_in, x_out, x_q, model): # 1. Tile the example pair in the top row and the query bottom-left [§4.2] canvas = grid_2x2(x_in, x_out, x_q) # (224, 224, 3) # 2. Hide the bottom-right cell and predict its visual codes [§3.1] [§3.2] codes = model.predict_codes(canvas, hide="bottom_right") # 3. Decode the codes to pixels; the answer is the bottom-right tile [§3.1] return decode(codes)[bottom_right_tile] # (111, 111, 3) ``` ```pseudocode # Stitched pair prompt: paint the query's output half [arxiv:2212.02499 §3.3] # Input: solved prompt pair (p_in, p_out) and query q, each (H, W, 3) # Output: predicted output for q, resized to q's original size def stitched_prompt(p_in, p_out, q, model): # 1. Stack the prompt pair above the query [§3.3] inputs = stack_rows(p_in, q) # (2H, W, 3) # 2. The query's output half is a masked placeholder; the model paints it [§3.3] targets = stack_rows(p_out, p_out) # bottom half is not used mask = bottom_half(inputs) # 1 where the model paints painted = model(inputs, targets, mask) # (2H, W, 3) # 3. Keep the painted query half and resize to the query's size [§3.3] return resize(painted[H:], size_of(q)) # (H_q, W_q, 3) ``` ```pseudocode # Learn one additive pixel prompt for a frozen CLIP-style model # [arxiv:2203.17274 §3.2.1] [eq. 1] # Input: frozen image encoder F, class text embeddings T, labelled set D, # prompt size p # Output: prompt v of the input's shape, shared by all test images def learn_pixel_prompt(F, T, D, p): # 1. Start from a padding frame of width p [§3.2.2]; init not stated v = padding_frame(p) # (C, H, W) for x, y in D: # 2. Add the prompt, then score against the class texts [§3.3] logits = F.image_scores(x + v, T) # (B, K) # 3. Cross-entropy on the correct class; only v gets a gradient [eq. 1] loss = cross_entropy(logits, y) # 4. One SGD step at learning rate 40 [§3.4] v = sgd_step(v, grad(loss, v), lr=40) return v def predict(F, T, v, x): # 5. The same v is added to every test image [eq. 2] return argmax(F.image_scores(x + v, T)) ``` ```pseudocode # One dual-stream layer with a bi-directional adapter [arxiv:2312.10611 §3.2] # Input: rgb, tir: tokens of the previous layer, (B, L, d) # Output: rgb, tir: tokens of the next layer, (B, L, d) def dual_stream_layer(rgb, tir): # 1. Attention stage: own output plus a prompt from the other stream [eq. 5] rgb_a = rgb + attn(norm(rgb)) + adapter_a(norm(tir)) tir_a = tir + attn(norm(tir)) + adapter_a(norm(rgb)) # 2. MLP stage: the same pattern with a second adapter [eq. 6] [eq. 7] rgb_o = rgb_a + mlp(norm(rgb_a)) + adapter_m(norm(tir_a)) tir_o = tir_a + mlp(norm(tir_a)) + adapter_m(norm(rgb_a)) # 3. Pass both streams to the next layer [eq. 3] return rgb_o, tir_o ``` ```pseudocode # Replace background patch tokens with learned prompts [arxiv:2210.04150 §3.3] # Input: patches (N_p, E); keep (N_p, 1) in {0, 1}, 1 where any pixel is # foreground; prompt (N_p, E), learned # Output: tokens (N_p, E), fed to the transformer blocks def mask_prompt_tokens(patches, keep, prompt): # 1. Keep foreground patches; swap background patches for prompt tokens [§3.3] return patches * keep + prompt * (1 - keep) ``` ## Contradictions - **Marks against coordinates.** In a 7B ablation the authors report prompts ahead of textual coordinates on PointQA (70.86 against 61.4) and on ViP-Bench with boxes (45.5 against 42.6). On Visual7W the coordinates are slightly ahead (86.36 against 86.04) [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. In the ViP-Bench table, LLaVA-1.5-7B with coordinates scores 44.8 on the synthetic set, against 41.6 with visual prompts [arxiv:2312.00784 §6]. The paper does not reconcile these; the results depend on the task and the model. - **Visual7W figures in one paper.** ViP-LLaVA gives 86.60 for the 7B model in its main table and 86.09 in its GPT-4V appendix. The 13B figures are 87.91 and 88.28 [arxiv:2312.00784 §5.2] [arxiv:2312.00784 app. Effect of Optional GPT-4V Region-Level Instruction Data]. Use the main table, which the abstract cites. - **Painter's masking ratio.** The paper gives 75% block-wise masking for training [arxiv:2212.02499 §3.2]. The released training script masks 784 of 1568 patches, which is 50%, as recorded on the [Painter card](/papers/2212.02499). The paper does not reconcile the two. - **Pair-prompt numbers across benchmarks.** Painter reports 62.3 mIoU for itself and 58.3 for MAE-VQGAN on FSS-1000 [arxiv:2212.02499 tab. 4]. MAE-VQGAN reports 27.17 on Pascal-5i [arxiv:2209.00647 §6]. These are different benchmarks, so they do not rank the two methods. - **Bi-directional against one-way.** BAT-Dual reaches 56.4 SR on LasHeR, above the full BAT at 56.3 [arxiv:2312.10611 tab. 2]. The full model still has the best PR in that table, 70.2 against 69.6 for BAT-Dual. ## Papers - [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](/papers/2312.00784): draws eight mark types onto the image and trains a LLaVA-style model to read them. - [End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering](/papers/2411.05755): draws numbered arrows for collision-free directions and lets a zero-shot VLM pick one each step. - [Visual Prompting via Image Inpainting](/papers/2209.00647): tiles example pairs and a query into a grid and fills the empty cell with a masked autoencoder. - [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](/papers/2212.02499): stacks a solved input-output pair above the query and paints the query's output half with one ViT-L model. - [Exploring Visual Prompts for Adapting Large-Scale Models](/papers/2203.17274): learns one additive frame for a frozen CLIP or vision model and adds it to every image. - [Bi-directional Adapter for Multi-modal Tracking](/papers/2312.10611): bi-directional adapters let RGB and thermal streams prompt each other in a frozen ViT tracker. - [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](/papers/2210.04150): learns prompt tokens for background patches so a CLIP classifier can label masked regions.