Exploring Visual Prompts for Adapting Large-Scale Models

In plain words

A large image model is usually adapted to a new job by retraining some of its weights, or by training a new output layer on top of it. Both need access to the model's inside. This paper asks whether you can leave the model completely alone and change only what you feed it. The method learns one picture-sized overlay of coloured pixels, a sort of frame, that is added to every input. The model never changes; only the frame is trained.

The frame turns out to work well with CLIP, a model that matches images to text. With CLIP the frame helps a lot over a plain text prompt and comes close to a classic linear probe, and it stays close to one when the test images come from a different source than the training images. With image classifiers that were not trained on text, the frame is far behind a linear probe. Because the frame is the only thing that changes, it can be handed to someone who has only input access to a model.

TL;DR

• Learns one additive visual prompt, applied to all images, while the pre-trained model stays frozen §3.2.1.
• The default is a padding frame of width 30 on 224×224 inputs, trained with SGD at learning rate 40 for 1,000 epochs §3.2.2 §3.4.
• Author-reported: on CLIP ViT-B/32 the prompt lifts the 12-dataset average from 54.2 (text prompt) to 78.2, against 83.1 for a linear probe tab. 1.
• It is weaker on ImageNet-pretrained classifiers and on BiT-M, where linear probes lead by about 40 points tab. 4.
• Code is released under MIT by the first author; several settings the paper leaves open are fixed in code and differ from the paper's wording main_clip.py.

Problem

Fine-tuning changes all parameters and linear probes need penultimate-layer activations; both need some access to the model §1 §2.4. Prompting in natural language adapts a frozen model through its input, but the authors note that pixel input has no discrete vocabulary to hand-write prompts from §3.2. The paper argues input-side adaptation is useful when the model is only reachable through an API or cannot be modified §1 §7. It does not state a compute budget or a dataset-size threshold where the method stops being useful.

Method

The method has two parts: an input transformation and an output transformation §3. For vision models, the authors say visual prompting is essentially the same as adversarial reprogramming; the v2 change log removes the earlier novelty claims app. A.3.

Input transformation (learn one prompt). Given a frozen model F and a labelled downstream set, learn one prompt v of the input's shape. During training the frozen model is asked for the likelihood of the correct label given x + v, and only v is updated §3.2.1 eq. 1.

Test time. The same v is added to every test image, and the prompted images go through F unchanged §3.2.1 eq. 2. The prompt is input-agnostic: one vector per dataset, not one per image §2.4.

Templates. Three templates are compared: a pixel patch at a random location, a pixel patch at a fixed location, and a padding frame. The number of parameters is C·p² for a patch and 2Cp(H + W − 2p) for padding, where C, H, W are channels, height and width and p is the prompt size §3.2.2. Padding with p = 30 is the default for all experiments §3.2.2 §6.2.

Resizing. Every downstream image is resized to the pre-trained model's input size, and the prompt is added directly to the input region, not padded around a smaller image §3.2.1. Images are preprocessed as in the model's own evaluation setting, and the paper calls this important §3.4.

Output transformation (map outputs to labels).

• CLIP: each class name is written into the fixed text prompt. Cosine similarities between the prompted image and each class text are normalised with a softmax, and the highest-probability class is the answer §3.3. Only the visual prompt is optimised; the text prompt is fixed §3.3.
• Vision models without text: downstream class indices are assigned arbitrarily to pre-trained output indices, and unassigned indices are dropped from the loss §3.3.

Training recipe (CLIP and vision). SGD at learning rate 40, with cosine decay over 1,000 epochs §3.4. Batch sizes are 256 for CLIP, and 128 (BiT-M, RN50) or 32 (Instagram) for the vision models §3.4. Images are 224×224 §3.4. The paper states no adversarial constraint on the perturbation; imperceptibility is not a goal §3.2.1.

# Learn one visual prompt for a frozen model  [§3.2.1] [eq. 1]
# Input:  frozen model F; labelled train set D; prompt size p
# Output: prompt v, one tensor shared by all images
def learn_visual_prompt(F, train_set, p):
    # Start from random values; the paper does not state the init
    # [code: models/prompters.py#L13-L16]
    v = padding_prompt_init(p, C, H, W)           # (1, C, H, W)
    # Only v is stepped; the weights of F are never updated  [§3.2.1]
    # SGD, momentum 0.9, no weight decay  [code: main_clip.py#L40-L46]
    optimizer = sgd(params=[v], lr=40, momentum=0.9)
    for step, (x, y) in batches(train_set, size=256):
        # Learning rate: linear warmup, then cosine decay  [§3.4]
        # [code: utils.py#L34-L48]
        set_learning_rate(optimizer, lr_schedule(step))
        # Add the prompt after the model's normalisation, not to raw pixels
        # [code: main_clip.py#L261]
        x_prompted = x + v                        # (B, C, H, W)
        # Scores from the frozen model; CLIP uses cosine similarity to text
        # embeddings of the class names  [§3.3]
        scores = F(x_prompted)
        # Cross-entropy; minimising it maximises the label likelihood [eq. 1]
        loss = cross_entropy(scores, y)
        loss.backward()
        optimizer.step()
    return v


# Test time: the same prompt is added to every test image  [§3.2.1] [eq. 2]
def predict(F, v, x_test):
    # The prompt is one fixed tensor for the whole dataset  [§3.2.1]
    return argmax(F(x_test + v))

Hyperparameters

Results

Author-reported. Setup: four frozen pre-trained models tab. 3; 12 image classification datasets §4.1 tab. 6; baselines are zero-shot text prompting, linear probe and full fine-tuning §4.2; WILDS distribution-shift datasets for robustness §4.1. Hardware and run time are not stated.

Full CLIP results, 12 datasets tab. 1. Shaded in the paper where the visual prompt beats the linear probe: EuroSAT, SVHN and CLEVR.

CLIP, WILDS distribution shift tab. 2. Shaded where the visual prompt beats the linear probe: Camelyon17.

Vision models, 12 datasets tab. 4. Shaded where the visual prompt beats the linear probe: SVHN and CLEVR for Instagram.

Vision models, WILDS tab. 5. Shaded where the visual prompt beats the linear probe and fine-tuning: Camelyon17 for Instagram.

Pre-trained models tab. 3.

Downstream datasets and CLIP text prompts tab. 6. Splits for OxfordPets, Flowers102, Food101, SUN397, DTD, EuroSAT and UCF101 follow a cited prior split source; the rest use the official splits app. A.2.

Other reported results, in text.

• Average gain of the CLIP visual prompt over the text prompt is 24 points §5.1 (computed as 78.2 − 54.2 tab. 1).
• The CLIP gap to linear probe is reduced to 4.5 points and to fine-tuning 3.5 points on WILDS averages §5.2 tab. 2.
• Dataset-level gain ranges from +83.2% (best) to −1% (worst) §6.1.
• Prompt-template ablation on EuroSAT with frozen CLIP: fixed-location templates beat random-location ones, and padding with p = 30 is best overall §6.2 fig. 6. A single-pixel prompt gives about +3% over the text prompt on EuroSAT §6.2 fig. 7.
• Output-mapping test with a dogs-vs-cats subset on RN50: 100% with semantically similar index mapping vs 62.5% with swapped indices §6.3.
• Lower-quality CLIP text prompts show larger visual-prompt gains §6.3 fig. 8.
• Model-architecture ablation on CIFAR100 is in fig. 9 (app. A.1); no numbers are given in text.

Paper vs code

Code: hjbahng/visual_prompting at 679e71f, MIT, copyright 2022 Hyojin Bahng LICENSE. It is linked from the project page and the abstract, and its owner is the first author. The repository has the CLIP and vision training scripts and the prompt module. It contains CIFAR-100 only.

• Dataset coverage. The paper reports 12 datasets and three WILDS sets §4.1. The code loads CIFAR-100 only, hard-coded in both scripts main_clip.py main_vision.py. The other datasets, the WILDS sets and the CLEVR/Camelyon templates are not in the release tab. 6.
• Text template. The paper uses per-dataset templates §4.2 tab. 6. The CLIP script fixes "This is a photo of a {}" for every run main_clip.py.
• Model selection. The paper does not say which checkpoint it reports. The code validates after every epoch on the CIFAR-100 test split, keeps the checkpoint with the best accuracy, and saves it as model_best main_clip.py main_clip.py. Any reported number taken from model_best was selected on the test split.
• Vision label mapping. The paper maps downstream classes arbitrarily, then drops unassigned indices §3.3. The code uses the first N output indices in order (class i to index i), with N = 100 for CIFAR-100 main_vision.py main_vision.py main_vision.py. That is not the semantic mapping the paper's own dog/cat test relies on §6.3.
• CLIP architecture flag. The paper's CLIP model is ViT-B/32 tab. 3. The code loads ViT-B/32 directly, and the --arch option is parsed but never read main_clip.py main_clip.py.
• Optimizer details. Momentum 0.9, zero weight decay and 1000 warmup steps are in the code only main_clip.py utils.py.
• Logit scale. The CLIP script clamps the logit scale to ln 100 after each step. The code comment attributes this to the original CLIP work main_clip.py. The paper does not mention it.
• Mixed precision. CLIP training uses autocast and a gradient scaler main_clip.py main_clip.py. Vision training is fp32 main_vision.py.
• Frozen model. Both scripts call model.eval() but never set requires_grad to false on the frozen model's parameters main_clip.py main_vision.py. Only the prompt is in the optimizer, so the frozen weights do not change, but backward still computes their gradients.
• Random-patch location. The paper says the random template is placed at a random location, without detail §3.2.2. The code draws one location per forward call, so one location is shared by the whole batch. It also draws a new location at evaluation time prompters.py.
• Fixed-patch location. The paper says "fixed location" without naming it §3.2.2. The code places the patch in the top-left corner prompters.py.
• Prompt init. Padding values are standard-normal draws in the normalised input space prompters.py. The paper gives no init.
• Zero-valued centre. The centre of the padding prompt is zero in the normalised space, so it equals each model's mean colour, not black prompters.py main_vision.py.
• Batch size for vision. The code default is 128 main_vision.py. The README says to use 32 for Instagram README.md, which matches §3.4.
• Baselines. The zero-shot text number on CIFAR-100 can be checked from the "Original Acc@1" line the CLIP script prints main_clip.py main_clip.py. The paper gives the same number as 63.1 tab. 1. Linear probe and fine-tuning baselines are not in the code.
• Early stopping. The patience default of 1000 means it is effectively never triggered main_clip.py main_clip.py.
• Text vs paper claim. The text says the vision-model gap is about +40% §5.1. From the averages in tab. 4 the gap is 39.3 points for Instagram, 40.0 for BiT-M and 46.8 for RN50 (our arithmetic).

No discrepancies in the training objective or in the schedule's base learning rate, epoch count and batch sizes.

Pitfalls

• Where the prompt is added. The paper says "image" §3.2.1. The code adds it after the model's normalisation, so the prompt values are in normalised units, not pixel units main_clip.py main_vision.py. A reimplementation that adds raw-pixel values gets a different scale for the same numbers.
• Preprocessing must match the model. The paper says closely following each model's evaluation preprocessing matters §3.4. CLIP uses its own preprocessing main_clip.py. Vision models use resize 256, crop 224 and ImageNet mean/std main_vision.py.
• Model selection on test. As above, reported CIFAR-100 accuracy from the released script is the best epoch on the test split main_clip.py main_clip.py.
• Class mapping for vision models. The default is index-order mapping. The paper's own ablation finds that semantically similar index assignments matter §6.3. The paper does not state which indices it used for its vision tables §3.3.
• CLIP logit scale. The CLIP loss depends on the logit scale used in the text-classifier output (clamped to ln 100 in code main_clip.py). The paper's equation does not show a scale term eq. 1 §3.3.
• Learning rate depends on the schedule. Both paper and code use 40 §3.4 main_clip.py. The paper names only a cosine decay; the warmup and its length come from code utils.py. A run with a different warmup is not the paper's run.
• Cost. The default run is 1,000 epochs and validates after every epoch with two forward passes, one with the prompt and one without main_clip.py main_clip.py. Gradients flow through the frozen model every step main_clip.py.
• Data scope. The WILDS robustness claims §5.2 and the other 11 dataset numbers cannot be reproduced from the release, which has only CIFAR-100 main_clip.py.
• Prompt meaning. The paper says the prompt need not be readable by people §3.2. It reports no analysis of what a learned frame encodes.
• Single-pixel claim. The single-pixel result on EuroSAT §6.2 has no separate code path. The nearest is the fixed-patch template with prompt_size=1 prompters.py. The paper does not give the pixel's location.
• Version history. v1 claimed methodological novelty; v2 withdrew it after the adversarial reprogramming overlap was pointed out app. A.3. This card follows v2.
• No adversarial constraint. The perturbation is not kept small or invisible §3.2.1. Results do not transfer to adversarial robustness claims.
• Hardware and time are not stated; compute cost cannot be estimated from the paper.

Lineage

• Builds on: adversarial reprogramming, which the paper cites as [elsayed2018adversarial] and says shares the mechanism §1 §2.3; soft prompt and prefix tuning in language models, cited as [lester2021power] and [li2021prefix] §2.1.
• Related concurrent work: Visual Prompt Tuning, which uses deep prompts in Vision Transformers §2.2.
• Compared against: fine-tuning, linear probe and zero-shot text prompting §4.2.
• Pre-trained models: CLIP [radford2021learning] tab. 3; Instagram ResNeXt [mahajan2018exploring]; BiT-M [kolesnikov2020big]; RN50 [he2016deep] tab. 3.
• Superseded by: none recorded in this wiki.