Visual prompting

Overview

The term covers five mechanisms, and the cards use it differently.

• Drawn marks. The prompt is a mark such as a box, ellipse, arrow, scribble or mask, drawn onto the image pixels before the image is encoded 2312.00784 §3.1. The encoder sees the mark as ordinary picture content. The mark can point at a region the user chose, or label a candidate action the system generated 2411.05755 §3.3.
• Example grids. The prompt is a picture of worked examples. An example input, its output and a query are tiled into one grid, and a masked image model fills the empty cell with the answer 2209.00647 §3.2. A person supplies the examples, and no weights change 2209.00647 §1. Painter uses the same idea with a model trained on this format: a solved pair is stacked above the query, and the query's output half is painted 2212.02499 §3.3.
• Learned prompt features. The prompt is a feature vector computed inside the network. A small adapter sends features from one stream into the other at each layer 2312.10611 §3.2. A user never draws anything.
• Learned prompt tokens. A binary mask marks the image patches that are pure background. Those patches are replaced by trainable tokens, and the tokens are the only prompt 2210.04150 §3.3. The mask comes from the segmenter's proposals, not from a person.
• Learned pixel overlay. The prompt is one tensor the size of the input. It is learned once per dataset from labelled examples, then added to every image, while the model stays frozen 2203.17274 §3.2.1. The paper says it need not be readable by people 2203.17274 §3.2.

Most of these steer a visual model that already exists. Painter is the exception: it trains one generalist model on the pair format, so the prompt format is part of training 2212.02499 §3.2. The mechanisms differ in who or what creates the prompt and where it enters the network. Whether a person must supply the prompt at inference time is the main constraint. ViP-LLaVA needs the user to point, and MAE-VQGAN needs example pairs from the user. VLMnav and OVSeg derive their marks or masks from the system, and BAT's prompts come from the other stream. Painter's prompt is a pair from its training set, chosen at random or by search, or a learned prompt 2212.02499 §3.3. The pixel overlay is trained once from labelled data and needs no input from a user or the system at inference 2203.17274 §3.2.1.

Variants

Drawn marks on the image

ViP-LLaVA draws one of eight mark types onto the image: rectangle, ellipse, point, scribble, triangle, mask, mask contour and arrow. The mark is alpha-blended in, and only the pixels under it change [arxiv:2312.00784 §3.1] 2312.00784 eq. 1. Colour, thickness and alpha are randomised in training 2312.00784 §5.1. The composite image goes through a LLaVA-style model with no region module 2312.00784 §3.1.

Training uses 520k region-prompt instruction pairs built on existing region-annotated datasets, among them RefCOCOg, PointQA, Visual Genome, Flickr30k Entities, VCR and Visual7W 2312.00784 §3.2. An optional 13k examples were written with GPT-4V from marked images 2312.00784 §3.3. The authors report 87.91% on Visual7W and 71.77% on PointQA-LookTwice for the 13B model 2312.00784 §5.2.

VLMnav uses drawn marks for a different job: choosing a direction. A depth-based check proposes collision-free directions, each is drawn as a numbered red arrow on the RGB image, and one VLM call picks a number 2411.05755 §3.1 2411.05755 §3.2 2411.05755 §3.3 2411.05755 §3.4. The VLM is used without fine-tuning [arxiv:2411.05755 §3]. On ObjectNav the authors report 50.4% success with the navigability proposals, 33.2% with static arrows and 29.8% with no arrows [arxiv:2411.05755 tab. 1]. Most of the gain over no arrows therefore comes from choosing the arrows well, not from drawing them. This is one benchmark, with a simulator that allows sliding along walls; without sliding the authors report 12.9% 2411.05755 tab. 1.

Example grids for an inpainting model

MAE-VQGAN is a masked autoencoder whose decoder predicts VQGAN codes. The example input and output and the query go into a 2x2 grid, the bottom-right cell is hidden, and the model predicts its codes [arxiv:2209.00647 §3.2] 2209.00647 §4.2. The model is pretrained on 88,645 arXiv computer-vision figures 2209.00647 §3.3. With one example pair, the authors report 27.17 mean mIoU on Pascal-5i foreground segmentation, against 14.91 for copying the example output 2209.00647 §6.

Pretraining matters. With ImageNet alone, the same model reaches 2.22 to 7.07 on the four segmentation splits, against 24.25 to 30.44 with the figures 2209.00647 tab. 1. In the few-shot table, the one-example grid reaches 32.5 on split 0, against 51.3 for FWB, which uses the full labelled base set and a segmentation-specific architecture 2209.00647 tab. 3.

Stitched pair prompts for a generalist model

Painter writes the output of each dense task as an RGB image. Depth, segmentation, keypoints and restoration outputs all become pictures 2212.02499 §3.1. It then trains one ViT-L model to paint a masked output half, given two stacked input-output pairs from the same task 2212.02499 §3.2. At test time, a solved pair is stacked above the query, and the query's output half is painted 2212.02499 §3.3. No task head is added, and a new task needs only new pairs in training 2212.02499 §3.1.

Results are author-reported. Painter beats the best listed specialist on NYUv2 depth (RMSE 0.288 against 0.330). It trails the specialist on ADE-20K segmentation (49.9 mIoU against 57.7), COCO panoptic (PQ 43.4 against 57.8) and COCO keypoints (AP 72.1 against 77.2) 2212.02499 tab. 1. Training on all tasks together does better than per-task training on most metrics, with keypoint AP and SIDD SSIM as the exceptions 2212.02499 tab. 2. For learned prompts, the model stays frozen and only the prompt tensors are trained 2212.02499 §3.3. On ADE-20K, random, searched and learned prompts give 49.3, 49.9 and 49.9 mIoU. The paper reports no variance, so that gap may be noise 2212.02499 tab. 3.

Learned pixel overlay for a frozen model

Exploring Visual Prompts learns one additive prompt the size of the input and adds it to every image. The model is frozen, and the prompt is trained by minimising the cross-entropy of the correct label 2203.17274 §3.2.1 2203.17274 eq. 1. Three templates are compared: a pixel patch at a random location, a patch at a fixed location, and a padding frame. The padding frame of width 30 on 224×224 inputs is the default 2203.17274 §3.2.2 2203.17274 §6.2. Training uses SGD at learning rate 40 for 1,000 epochs 2203.17274 §3.4.

For CLIP, class names go into a fixed text template, and only the image prompt is optimised 2203.17274 §3.3. On 12 datasets the authors report 78.2 for the prompt, against 54.2 for the text prompt alone and 83.1 for a linear probe 2203.17274 tab. 1. On CLIP over three WILDS sets the prompt averages 58.1, against 62.6 for the linear probe, but it is ahead on Camelyon17, 89.8 against 84.9 2203.17274 tab. 2. For the Instagram, BiT-M and RN50 vision models, which have no text head, the prompt reaches 25.7 to 32.6 on the same 12 datasets, against 71.7 to 72.6 for linear probes 2203.17274 tab. 4. The paper gives no adversarial bound on the perturbation, so these results say nothing about robustness [arxiv:2203.17274 §3.2.1].

The card's Paper vs code section records that the released code loads only CIFAR-100, so the other datasets cannot be reproduced from it. The same section notes that the released script picks its checkpoint on that test split (Exploring Visual Prompts).

Learned prompt features across two streams

BAT tracks an object from an RGB camera and a thermal camera. It builds on a frozen OSTrack ViT-B/16 2312.10611 §4.1.2. In every layer, a bi-directional adapter adds to each stream a prompt computed from the other stream, once in the attention stage and once in the MLP stage 2312.10611 §3.2 2312.10611 eq. 5. Only the adapters are trained, 0.32M parameters in total 2312.10611 §1. The authors report PR 70.2 and SR 56.3 on LasHeR, and MPR 86.8 and MSR 64.1 on RGBT234 [arxiv:2312.10611 tab. 1]. The earlier prompt-based tracker ViPT reports 65.1 and 52.5 on LasHeR and 83.5 and 61.7 on RGBT234 in the same table 2312.10611 tab. 1.

Learned prompt tokens in masked patches

OVSeg uses mask prompt tuning in a two-stage open-vocabulary segmenter. Each proposal is cropped, its background pixels are blanked, and every patch that is entirely background is replaced by a learned token. Partly covered patches keep their image tokens, because their boundary pixels still matter 2210.04150 §3.3. The prompts enter at the input and at deeper layers. The default prompt depth is 3 2210.04150 §4.1 2210.04150 app. A.5.

The prompts can be trained with CLIP frozen, starting from the original weights, or on top of a CLIP that has first been fine-tuned 2210.04150 §4.1. On ADE20K-150 the author-reported mIoU is 21.8 for the baseline, 26.5 with prompts only, 28.8 with full fine-tuning only and 29.6 with both [arxiv:2210.04150 tab. 3]. Against deep visual prompt tuning, the appendix reports 26.5 against 25.5 on the same dataset, with the VPT variant using more tokens and about 40% more compute 2210.04150 tab. 7.

When to use which

Implementation sketch

The variants share no code. The first block shows how a mark enters the image; the second shows the MAE-VQGAN example grid; the third shows Painter's stitched pair; the fourth shows the pixel overlay; the fifth shows the adapter layer; the sixth shows the prompt tokens. Paper and released code differ on several details, such as the alpha range, CLIP layer indices, adapter normalisation, grid padding, mask ratio, prompt learning rates, where the overlay is added relative to normalisation, and the class-index mapping for models without text. Check each card's "Paper vs code" section before copying any sketch.

# Draw one visual prompt onto the image  [arxiv:2312.00784 §3.1] [eq. 1]
# Input:  image (H, W, 3); mark rasterised as RGBA (H, W, 4); alpha in [0, 1]
# Output: composite image (H, W, 3), fed to the usual vision encoder

def overlay_prompt(image, mark, alpha):
    # 1. Rasterise the mark: box, ellipse, arrow, scribble or mask  [§3.2]
    layer = rasterise(mark)                       # (H, W, 4)
    # 2. Blend only the pixels the mark covers  [eq. 1]
    return blend(image, layer, alpha)             # (H, W, 3)

# Visual prompt as an example grid, read from a masked image model
# [arxiv:2209.00647 §3.2] [§4.2]
# Input:  example pair (x_in, x_out) and query x_q, each a 111 x 111 tile
#         [arxiv:2209.00647 tab. 3]
# Output: predicted output tile for x_q, (111, 111, 3)

def grid_prompt(x_in, x_out, x_q, model):
    # 1. Tile the example pair in the top row and the query bottom-left  [§4.2]
    canvas = grid_2x2(x_in, x_out, x_q)           # (224, 224, 3)
    # 2. Hide the bottom-right cell and predict its visual codes  [§3.1] [§3.2]
    codes = model.predict_codes(canvas, hide="bottom_right")
    # 3. Decode the codes to pixels; the answer is the bottom-right tile  [§3.1]
    return decode(codes)[bottom_right_tile]       # (111, 111, 3)

# Stitched pair prompt: paint the query's output half  [arxiv:2212.02499 §3.3]
# Input:  solved prompt pair (p_in, p_out) and query q, each (H, W, 3)
# Output: predicted output for q, resized to q's original size

def stitched_prompt(p_in, p_out, q, model):
    # 1. Stack the prompt pair above the query  [§3.3]
    inputs = stack_rows(p_in, q)                   # (2H, W, 3)
    # 2. The query's output half is a masked placeholder; the model paints it  [§3.3]
    targets = stack_rows(p_out, p_out)             # bottom half is not used
    mask = bottom_half(inputs)                     # 1 where the model paints
    painted = model(inputs, targets, mask)         # (2H, W, 3)
    # 3. Keep the painted query half and resize to the query's size  [§3.3]
    return resize(painted[H:], size_of(q))         # (H_q, W_q, 3)

# Learn one additive pixel prompt for a frozen CLIP-style model
# [arxiv:2203.17274 §3.2.1] [eq. 1]
# Input:  frozen image encoder F, class text embeddings T, labelled set D,
#         prompt size p
# Output: prompt v of the input's shape, shared by all test images

def learn_pixel_prompt(F, T, D, p):
    # 1. Start from a padding frame of width p  [§3.2.2]; init not stated
    v = padding_frame(p)                          # (C, H, W)
    for x, y in D:
        # 2. Add the prompt, then score against the class texts  [§3.3]
        logits = F.image_scores(x + v, T)          # (B, K)
        # 3. Cross-entropy on the correct class; only v gets a gradient  [eq. 1]
        loss = cross_entropy(logits, y)
        # 4. One SGD step at learning rate 40  [§3.4]
        v = sgd_step(v, grad(loss, v), lr=40)
    return v

def predict(F, T, v, x):
    # 5. The same v is added to every test image  [eq. 2]
    return argmax(F.image_scores(x + v, T))

# One dual-stream layer with a bi-directional adapter  [arxiv:2312.10611 §3.2]
# Input:  rgb, tir: tokens of the previous layer, (B, L, d)
# Output: rgb, tir: tokens of the next layer, (B, L, d)

def dual_stream_layer(rgb, tir):
    # 1. Attention stage: own output plus a prompt from the other stream  [eq. 5]
    rgb_a = rgb + attn(norm(rgb)) + adapter_a(norm(tir))
    tir_a = tir + attn(norm(tir)) + adapter_a(norm(rgb))
    # 2. MLP stage: the same pattern with a second adapter  [eq. 6] [eq. 7]
    rgb_o = rgb_a + mlp(norm(rgb_a)) + adapter_m(norm(tir_a))
    tir_o = tir_a + mlp(norm(tir_a)) + adapter_m(norm(rgb_a))
    # 3. Pass both streams to the next layer  [eq. 3]
    return rgb_o, tir_o

# Replace background patch tokens with learned prompts  [arxiv:2210.04150 §3.3]
# Input:  patches (N_p, E); keep (N_p, 1) in {0, 1}, 1 where any pixel is
#         foreground; prompt (N_p, E), learned
# Output: tokens (N_p, E), fed to the transformer blocks

def mask_prompt_tokens(patches, keep, prompt):
    # 1. Keep foreground patches; swap background patches for prompt tokens  [§3.3]
    return patches * keep + prompt * (1 - keep)

Contradictions

• Marks against coordinates. In a 7B ablation the authors report prompts ahead of textual coordinates on PointQA (70.86 against 61.4) and on ViP-Bench with boxes (45.5 against 42.6). On Visual7W the coordinates are slightly ahead (86.36 against 86.04) [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. In the ViP-Bench table, LLaVA-1.5-7B with coordinates scores 44.8 on the synthetic set, against 41.6 with visual prompts 2312.00784 §6. The paper does not reconcile these; the results depend on the task and the model.
• Visual7W figures in one paper. ViP-LLaVA gives 86.60 for the 7B model in its main table and 86.09 in its GPT-4V appendix. The 13B figures are 87.91 and 88.28 2312.00784 §5.2 [arxiv:2312.00784 app. Effect of Optional GPT-4V Region-Level Instruction Data]. Use the main table, which the abstract cites.
• Painter's masking ratio. The paper gives 75% block-wise masking for training 2212.02499 §3.2. The released training script masks 784 of 1568 patches, which is 50%, as recorded on the Painter card. The paper does not reconcile the two.
• Pair-prompt numbers across benchmarks. Painter reports 62.3 mIoU for itself and 58.3 for MAE-VQGAN on FSS-1000 2212.02499 tab. 4. MAE-VQGAN reports 27.17 on Pascal-5i 2209.00647 §6. These are different benchmarks, so they do not rank the two methods.
• Bi-directional against one-way. BAT-Dual reaches 56.4 SR on LasHeR, above the full BAT at 56.3 2312.10611 tab. 2. The full model still has the best PR in that table, 70.2 against 69.6 for BAT-Dual.

Papers

• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts: draws eight mark types onto the image and trains a LLaVA-style model to read them.
• End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering: draws numbered arrows for collision-free directions and lets a zero-shot VLM pick one each step.
• Visual Prompting via Image Inpainting: tiles example pairs and a query into a grid and fills the empty cell with a masked autoencoder.
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning: stacks a solved input-output pair above the query and paints the query's output half with one ViT-L model.
• Exploring Visual Prompts for Adapting Large-Scale Models: learns one additive frame for a frozen CLIP or vision model and adds it to every image.
• Bi-directional Adapter for Multi-modal Tracking: bi-directional adapters let RGB and thermal streams prompt each other in a frozen ViT tracker.
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP: learns prompt tokens for background patches so a CLIP classifier can label masked regions.