Paper · 2021-10-16 · cs.CV, cs.CL
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
FewVLM is a T5-style encoder-decoder vision-language model (224M base, 740M large) pre-trained with prefix and masked language modeling, then adapted to VQA and captioning with 16 examples and a text prompt. Author-reported: 43.4 zero-shot VQAv2 for base versus 29.5 for Frozen (7B). Prompt wording strongly changes zero-shot accuracy but matters little with a few examples.
Few-shot learningPrompt engineeringVision-language pre-trainingMasked language modelingvision-languagefew-shotprompt-based-learningencoder-decoder
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Large vision-language models can answer questions about a picture or write a caption after seeing only a handful of examples. Their size makes them slow and expensive to run. This work trains a much smaller model that reads the picture as a set of object-region descriptions, reads the words as text, and writes its answer as text.
The model learns from image-caption data in two ways: it fills in hidden words, and it continues a sentence from its beginning. For a new task, the question is wrapped in a short fixed template, called a prompt, that tells the model what kind of answer to produce. The study finds that the exact wording of that template changes results a lot when there are no training examples, and much less once a few examples are available. All numbers are the authors' own, on standard benchmarks.
TL;DR
FewVLM is a generative vision-language model: an encoder-decoder that takes text plus 36 image-region features and writes the answer or caption as text §4.1. It is pre-trained on image-caption data with prefix and masked language modeling §4.2, then fine-tuned on 16 training examples with a hand-written prompt §3.1, §5. Author-reported, the base model (224M) beats Frozen (7B) on zero-shot VQAv2, 43.4 against 29.5 tab. 2. Prompt wording matters a lot for zero-shot accuracy and little for few-shot accuracy §6.5. The released code differs from the paper in several training details; see Paper vs code.
Problem
Few-shot learning with large language models is practical, but the vision-language few-shot models compared here are very large: Frozen has 7B parameters and PICa is based on GPT-3 (175B) §1, tab. 2. The authors want a moderate-sized model that can run on standard hardware and learn from very little labelled data §1.
The setting is fixed as follows. A zero-shot model sees no training or dev data. A few-shot model gets a training set and a dev set of equal size, both 16 examples, drawn from the training data and used to tune hyperparameters and pick the checkpoint §3.1. Evaluation uses VQAv2 validation, GQA test-dev, OK-VQA test, the Karpathy test split of Flickr30k, the NoCaps validation set, and 5-way miniImageNet §3.3, §3.4.
Three analysis questions are posed: how prompt design affects zero- and few-shot learning, whether prompt design still matters with more training data, and how the two pre-training objectives affect zero- and few-shot performance §3.2.
Method
Architecture §4.1, app. A. Each image is turned into 36 region features, produced by a detector trained on Visual Genome. The region features are appended to the text tokens in the encoder §4.1. The decoder generates target text. The base model follows the VL-T5 layout and the large model the T5 layout app. A, tab. 9.
Training objective §4.1, eq. 1. Training minimises the negative log-likelihood of the target tokens, given the input text and the image:
L_\theta = -\sum_{i=1}^{|y|} \log P_\theta(y_i \mid y_{<i}, x, v) eq. 1
Here x is the input text, v the image regions and y the target tokens. The model is not task-specific, so no task heads are added §4.1.
Pre-training §4.2, fig. 3.
- Data: MS COCO and Visual Genome image-caption data, 9.18M image-text pairs over 180K distinct images §4.2.
- Prefix language modeling (PrefixLM) follows T5. A caption is cut at a random point. The part before the cut goes to the encoder with the image, and the rest is the decoder target §4.2.
- Masked language modeling (MaskedLM) follows VL-T5. About 15% of the input words are replaced by numbered sentinel tokens, and the decoder generates the hidden spans §4.2.
Low-resource adaptation §5, fig. 2. A prompt template P turns an input and a label into input text x and target text y. The model is trained on these with the same loss eq. 1. At inference the same prompt is used, the model generates text, and the target template is removed to leave the label §5.
Prompt designs §5.1, tab. 1, tab. 11.
| ID | Task | Input text (paraphrased) |
|---|---|---|
| P1 | VQA | the question, then one sentinel |
| P2 | VQA | a "question: ... answer:" frame around the question |
| P3 | VQA | the P2 frame, followed by one sentinel |
| Q1 / Q2 / Q3 | Captioning | a fixed phrase meaning "a picture of", "a photo of" or "an image of" |
| noisy (ablation) | VQA | 5 irrelevant instructions, 5 strings of random vocabulary tokens, 5 unrelated captions tab. 11 |
| miniImageNet | 5-way classification | a fixed phrase ending in one sentinel |
VQA targets are either the answer alone or a sentinel followed by the answer. In the main experiments the target is the sentinel plus answer, and the input is P3 §6.1. For captioning, the target is the caption itself tab. 1. VQA is treated as generation, so answers are produced as strings, not chosen from a fixed answer list §5.1.
Pseudocode: pre-training examples §4.2. Each caption gives two examples, one per objective. Captions of two words or fewer are skipped for T5 backbones, and duplicate sentences are removed pretrain_data.py.
# Build two pre-training examples from one caption [§4.2]
# Captions of two words or fewer are skipped (code) [code: src/pretrain_data.py#L72]
def make_examples(caption, image_regions):
words = caption.split()
# PrefixLM: cut at one random word position [§4.2] [code: src/preprocess.py#L104]
cut = random_int(1, len(words) - 1)
prefix_example = {
"encoder_text": words[:cut],
"target_text": words[cut:],
"image": image_regions,
}
# MaskedLM: hide about 15% of words, at least one, in numbered spans
# [§4.2] [code: src/preprocess.py#L30]
masked_example = mask_spans(words, mask_ratio=0.15)
masked_example["image"] = image_regions
return [prefix_example, masked_example]Pseudocode: loss for one example eq. 1 pretrain_model.py.
# Loss for one example, following eq. 1 [eq. 1]
# The code divides the summed token loss by the number of target tokens
# [code: src/pretrain_model.py#L41]; eq. 1 has no such division
def example_loss(model, encoder_text, image, target_tokens):
# Negative log-likelihood of each target token [eq. 1]
token_nll = model.token_nll(encoder_text, image, target_tokens)
# Average over the target tokens of this example [code: src/pretrain_model.py#L41]
return token_nll.sum() / len(target_tokens)Pseudocode: few-shot split and VQA input §3.1, §5.1.
# Few-shot train and dev sets from one shuffled pool [§3.1]
# Code: seeded shuffle, disjoint 16-example slices [code: src/vqa_data.py#L95-L101]
def few_shot_split(pool, data_seed, num_data=16):
shuffled = seeded_shuffle(pool, data_seed)
train = shuffled[:num_data]
dev = shuffled[num_data:2 * num_data]
return train, dev# VQA input text with prompt P3 [§5.1] [code: src/vqa_data.py#L185]
# The whole input is cut to 20 tokens before image tokens are added
def vqa_input_text(question):
return "question: " + question + " answer: <extra_id_0>"# VQA target text: sentinel, then answer [§6.1] [code: src/vqa_data.py#L231]
# Cut to 10 tokens [code: src/vqa_data.py#L231]
def vqa_target_text(answer):
return "<extra_id_0> " + answerHyperparameters
Values marked "not stated" do not appear in the paper. Code values come from the official repository at code_commit.
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbone, base | 12+12 layers, width 768, FFN 3,072, 12 heads of size 64; 224M params | app. A tab. 9 tab. 2 | VL-T5 layout |
| Backbone, large | 24+24 layers, width 1,024, FFN 4,096, 16 heads of size 64; 740M params | app. A tab. 9 tab. 2 | T5 layout; the repo's scripts set only t5-base pretrain.sh |
| Image input | 36 region features per image | §4.1 param.py | Feature dimension 2048 in code param.py; not stated in paper |
| Pre-training data | 9.18M pairs, 180K images (MS COCO, Visual Genome) | §4.2 | Code trains on the COCO re-split train set and a Visual Genome set named vgnococo pretrain.sh |
| Pre-training batch | 1,280 (base), 800 (large) | §6.1 | Code: 320 per process pretrain.sh; see Paper vs code |
| Pre-training epochs | 30 | §6.1 | pretrain.sh |
| Pre-training learning rate | 1e-4 | §6.1 | pretrain.sh |
| Pre-training warm-up | 5% of steps, linear | §6.1 | pretrain.sh; linear decay in trainer_base.py (decay not stated in paper) |
| Pre-training optimiser | not stated | pretrain.sh | AdamW; eps 1e-6, betas 0.9 and 0.999, weight decay 0.01 param.py param.py |
| Pre-training gradient clip | not stated | pretrain.sh | 1.0 in code |
| Pre-training precision | not stated | pretrain.sh | Mixed precision (fp16) in code |
| Dropout | not stated | param.py | 0.1 default in code |
| MaskedLM mask rate | 15% of input words | §4.2 | At least one word; preprocess.py |
| Input text length | not stated | vqa_data.py | 20 tokens for VQA input; 20 for pre-training text param.py |
| Few-shot train and dev size | 16 and 16 | §3.1 §6.1 | Disjoint slices of one shuffled split vqa_data.py |
| Few-shot epochs | 200 | §6.1 §3.4 | VQA.sh |
| Few-shot learning rate | 5e-5 | §6.1 | VQA.sh |
| Few-shot warm-up | 5% | §6.1 | 10% in code VQA.sh |
| Few-shot batch size | not stated | 30 in code VQA.sh | |
| Few-shot gradient clip | not stated | 5 in code VQA.sh | |
| Few-shot optimiser | not stated | AdamW in code VQA.sh | |
| Checkpoint selection | best on dev set | §3.4 | VQA: highest dev accuracy vqa.py |
| Decoding | not stated | Beam search with 5 beams, up to 20 generated tokens in code VQA.sh param.py | |
| Prompt, VQA | P3, target sentinel plus answer | §6.1 | Code default is no prompt param.py; the flag --prompt 3 is needed VQA.sh |
| Prompt, captioning | "an image of" | §6.1 | Code needs --prefix image flickr.sh |
| Seeds and splits | five random train/dev splits, averaged | §3.4 | Code runs one data seed per run; default 9595 param.py; scripts use 42 VQA.sh |
Results
Author-reported, on the paper's own splits and evaluation sets §3.4. Model sizes are given as in the paper. Few-shot results are averages over five splits.
Zero-shot VQA tab. 2: accuracy.
| Model | Size | VQAv2 | OK-VQA | GQA |
|---|---|---|---|---|
| Unified VLP | 122M | 0.0 | - | - |
| VL-T5 (no VQA pre-training) | 224M | 13.5 | 5.8 | 6.3 |
| Frozen | 7B | 29.5 | 5.9 | - |
| PICa | 175B | - | 17.5 | - |
| FewVLM base | 224M | 43.4 | 11.6 | 27.0 |
| FewVLM large | 740M | 47.7 | 16.5 | 29.3 |
Few-shot VQA tab. 3: accuracy, 16 examples unless noted.
| Model | Size | VQAv2 | OK-VQA | GQA |
|---|---|---|---|---|
| Unified VLP | 122M | 24.3 | - | - |
| VL-T5 (no VQA pre-training) | 224M | 31.8 | 12.7 | 19.6 |
| Frozen (4 examples) | 7B | 38.2 | 12.6 | - |
| PICa (16 examples) | 175B | 54.3 | 43.3 | - |
| FewVLM base, 4 examples | 224M | 45.1 | 14.5 | 26.9 |
| FewVLM base | 224M | 48.2 | 15.0 | 32.2 |
| FewVLM large | 740M | 51.1 | 23.1 | 35.7 |
| Fully fine-tuned (reference) | - | 72.6 | - | 61.5 |
Zero-shot captioning tab. 4: CIDEr and SPICE.
| Model | Size | NoCaps CIDEr | NoCaps SPICE | Flickr30k CIDEr | Flickr30k SPICE |
|---|---|---|---|---|---|
| Unified VLP | 122M | - | - | 24.9 | 7.2 |
| VL-T5 (no VQA pre-training) | 224M | 4.4 | 5.3 | 2.6 | 2.0 |
| SimVLM (huge) | - | 101.4 | - | - | - |
| FewVLM base | 224M | 42.2 | 8.5 | 31.0 | 10.0 |
| FewVLM large | 740M | 47.7 | 9.1 | 36.5 | 10.7 |
Few-shot captioning tab. 5: CIDEr and SPICE, 16 examples.
| Model | Size | NoCaps CIDEr | NoCaps SPICE | Flickr30k CIDEr | Flickr30k SPICE |
|---|---|---|---|---|---|
| Unified VLP | 122M | - | - | 28.8 | 9.4 |
| VL-T5 (no VQA pre-training) | 224M | 22.0 | 6.8 | 12.8 | 8.3 |
| FewVLM base | 224M | 48.6 | 10.0 | 32.6 | 12.8 |
| FewVLM large | 740M | 53.1 | 10.4 | 37.0 | 13.5 |
| Fully fine-tuned (reference) | - | 112.2 | 13.1 | 67.4 | 17.0 |
5-way miniImageNet tab. 6: accuracy by shots per class.
| Model | 1 shot | 3 shots | 5 shots |
|---|---|---|---|
| Frozen (7B) | 14.5 | 34.7 | 33.8 |
| FewVLM base, no prompt | 48.0 | 75.0 | 82.6 |
| FewVLM base | 57.0 | 78.0 | 84.2 |
| FewVLM large | 57.1 | 78.3 | 84.4 |
| AFHN | 62.3 | - | 78.1 |
Prompt study, FewVLM base, zero-shot tab. 7: VQAv2 accuracy and Flickr30k CIDEr.
| Prompt | VQAv2 | Flickr30k |
|---|---|---|
| No prompt | 3.7 | 9.6 |
| P1 / Q1 ("a picture of") | 9.9 | 15.2 |
| P2 / Q2 ("a photo of") | 19.0 | 25.6 |
| P3 / Q3 ("an image of") | 43.4 | 31.0 |
Pre-training objectives, FewVLM base tab. 8: VQAv2, GQA and Flickr30k CIDEr.
| Objective | Zero-shot VQAv2 | Zero-shot GQA | Zero-shot Flickr30k | Few-shot VQAv2 | Few-shot GQA | Few-shot Flickr30k |
|---|---|---|---|---|---|---|
| MaskedLM | 42.4 | 25.1 | 4.6 | 46.0 | 31.4 | 18.5 |
| PrefixLM | 11.9 | 6.7 | 26.8 | 40.8 | 27.6 | 31.8 |
| MaskedLM + PrefixLM | 43.4 | 27.0 | 31.0 | 48.2 | 32.2 | 32.6 |
Appendix app. B, app. D, tab. 10, tab. 12. COCO captioning, zero-shot and few-shot CIDEr/SPICE: FewVLM base 84.5/16.2 and 98.7/18.9; FewVLM large 92.1/17.3 and 100.4/19.1; SimVLM (huge) 102.3/22.1 zero-shot tab. 10. Pre-training on Conceptual Captions instead of COCO and Visual Genome lowers few-shot VQAv2 from 48.2 to 36.7, GQA from 32.2 to 25.9 and Flickr30k CIDEr from 32.6 to 22.3 tab. 12.
Paper vs code
Checked against the official repository at code_commit 3c17bc980a0d28cfb06eecdcf2e13a991c09a00e. The repository has no LICENSE file, so no code is quoted here; only paths and line numbers are cited.
- Few-shot warm-up. Paper: 5% §6.1. Code: 0.1 in all five few-shot scripts VQA.sh flickr.sh COCOCaption.sh Nocaps.sh OKVQA.sh.
- Pre-training batch. Paper: 1,280 for base and 800 for large §6.1. Code: 320 per process with a per-process data loader pretrain.sh pretrain_data.py, so the total batch is 320 times the number of GPUs. The README example uses 8 GPUs, which would give 2,560 README.md. The GPU count behind the paper's 1,280 is not stated.
- Pre-training loss. Paper: eq. 1 sums the token losses eq. 1. Code: sums per example, then divides by the number of target tokens, then averages over the batch pretrain_model.py pretrain_model.py. The two differ in scale for examples of different lengths.
- Examples per caption. The paper describes two objectives §4.2. Code builds both examples from every caption, so each caption appears twice per epoch pretrain_data.py.
- Pre-training checkpoint. The paper does not say which checkpoint is used downstream. Code saves a checkpoint every epoch; the best-loss copy is commented out pretrain.py. The VQA script loads the last one,
Epoch30VQA.sh. - Default prompt. Paper: P3 for VQA and "an image of" for captioning §6.1. Code: the default is no prompt param.py and captioning uses an empty prefix when none is given flickr_data.py. The scripts' example commands pass
--prompt 3and--prefix image, but the scripts themselves do not VQA.sh flickr.sh. - Sentinel token. The paper writes
<text_1>§5.1. Code uses T5's<extra_id_0>vqa_data.py preprocess.py. This is a naming difference; behaviour should match, but it is not verified. - Few-shot seeds. Paper: average over five random splits §3.4. Code: one data seed per run, and the example commands use seed 42 VQA.sh. No script runs the five-split average.
- Decoding. The paper does not state a decoding method. Code uses beam search with 5 beams VQA.sh.
- Large model. The paper uses a T5 layout for the large model app. A. The scripts set only
t5-basepretrain.sh, so no large-model configuration is in the repository. - Not in the repository. The Faster R-CNN feature extractor, the pre-training image-caption data, and any miniImageNet pipeline are not in the repository. Features are read from precomputed HDF5 files vqa_data.py. The README points to VL-T5 for feature extraction and to a download folder for the data README.md, "Datasets".
- Training target for VQA. Not stated in the paper. Code draws the answer for each example at random, weighted by the answer scores, each time the example is loaded vqa_data.py.
No discrepancies were found for the learning rates (1e-4 for pre-training, 5e-5 for few-shot), the pre-training epochs (30), the few-shot epochs (200), or the 15% masking rate.
Pitfalls
- Set the prompt explicitly. The code default is no prompt param.py. Without a prompt, zero-shot VQAv2 is 3.7 against 43.4 with P3 tab. 7. Few-shot VQA and captioning need the same flag VQA.sh.
- Zero-shot is very sensitive to wording. On Flickr30k the three captioning phrases give CIDEr from 15.2 to 31.0 tab. 7. With noisy prompts, zero-shot accuracy is close to 0 §6.5.2. Few-shot results move much less §6.5.2, fig. 4.
- Truncation cuts the template. The whole prompt-plus-question string is cut to 20 tokens before image tokens are added vqa_data.py modeling_t5.py. For long questions, the trailing "answer" frame and the sentinel can be lost. The paper does not mention this.
- Answer targets are sampled. Each time a VQA example is loaded, its answer is drawn from the weighted answer list, so the target can change between epochs vqa_data.py. The paper does not state this.
- Few-shot variance is large. The paper averages five splits §3.4. The target prompt "[A]" alone has a larger spread across training sizes than the sentinel version §6.5.2, fig. 6. A single run from the scripts is one sample.
- Checkpoint choice. For VQA the best dev accuracy wins, and ties go to the later epoch vqa.py. Pre-training keeps every epoch and the scripts use the last one pretrain.py.
- Masking produces one sentinel per span. Adjacent masked words merge into one span, and the number of masked words is at least one preprocess.py preprocess.py. The number of sentinels is therefore at most the number of masked words.
- Prefix split is uniform over word positions. The cut is drawn uniformly from 1 to n−1 words preprocess.py, so a one-word caption cannot be split. The code skips captions of two words or fewer for T5 backbones pretrain_data.py.
- Abstract margin. The abstract's 18.2-point margin over Frozen matches the large model's zero-shot VQAv2 (47.7) minus Frozen (29.5) tab. 2. The base model's margin is 13.9 (derived from the same table). The text's "31 times" refers to model size §6.2.
- NoCaps has no training split in the paper. It uses COCO captioning training data §3.4; the script does the same Nocaps.sh.
- Cost. The paper reports no inference cost or latency. The code decodes with 5 beams VQA.sh, so reported speed does not transfer to greedy decoding.
- Version. Compiled from arXiv v2 (updated 2022-03-15). The licence was recorded by
wikillm fetchfrom the arXiv metadata. Other versions may differ.
Lineage
- Builds on: VL-T5 for the base layout and MaskedLM app. A §4.2; T5 for the large layout and the PrefixLM objective app. A §4.2; prompt-based few-shot work in NLP §1, §3.2.
- Compared against: Frozen, PICa, SimVLM, Unified VLP, VL-T5 without VQA pre-training, AFHN, and fully fine-tuned reference models §3.5 tab. 2–6.
- Superseded by: none known from content in this wiki.
Source and license
Adapted from “A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models” by Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren (arXiv:2110.08484), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/woojeongjin/FewVLM at commit 3c17bc980a0d (none (no LICENSE file in repository)). Any quoted code is shown under that license.