wikillm

Paper · 2021-10-16 · cs.CV, cs.CL

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

FewVLM is a T5-style encoder-decoder vision-language model (224M base, 740M large) pre-trained with prefix and masked language modeling, then adapted to VQA and captioning with 16 examples and a text prompt. Author-reported: 43.4 zero-shot VQAv2 for base versus 29.5 for Frozen (7B). Prompt wording strongly changes zero-shot accuracy but matters little with a few examples.

Few-shot learningPrompt engineeringVision-language pre-trainingMasked language modelingvision-languagefew-shotprompt-based-learningencoder-decoder

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Large vision-language models can answer questions about a picture or write a caption after seeing only a handful of examples. Their size makes them slow and expensive to run. This work trains a much smaller model that reads the picture as a set of object-region descriptions, reads the words as text, and writes its answer as text.

The model learns from image-caption data in two ways: it fills in hidden words, and it continues a sentence from its beginning. For a new task, the question is wrapped in a short fixed template, called a prompt, that tells the model what kind of answer to produce. The study finds that the exact wording of that template changes results a lot when there are no training examples, and much less once a few examples are available. All numbers are the authors' own, on standard benchmarks.

TL;DR

FewVLM is a generative vision-language model: an encoder-decoder that takes text plus 36 image-region features and writes the answer or caption as text §4.1. It is pre-trained on image-caption data with prefix and masked language modeling §4.2, then fine-tuned on 16 training examples with a hand-written prompt §3.1, §5. Author-reported, the base model (224M) beats Frozen (7B) on zero-shot VQAv2, 43.4 against 29.5 tab. 2. Prompt wording matters a lot for zero-shot accuracy and little for few-shot accuracy §6.5. The released code differs from the paper in several training details; see Paper vs code.

Problem

Few-shot learning with large language models is practical, but the vision-language few-shot models compared here are very large: Frozen has 7B parameters and PICa is based on GPT-3 (175B) §1, tab. 2. The authors want a moderate-sized model that can run on standard hardware and learn from very little labelled data §1.

The setting is fixed as follows. A zero-shot model sees no training or dev data. A few-shot model gets a training set and a dev set of equal size, both 16 examples, drawn from the training data and used to tune hyperparameters and pick the checkpoint §3.1. Evaluation uses VQAv2 validation, GQA test-dev, OK-VQA test, the Karpathy test split of Flickr30k, the NoCaps validation set, and 5-way miniImageNet §3.3, §3.4.

Three analysis questions are posed: how prompt design affects zero- and few-shot learning, whether prompt design still matters with more training data, and how the two pre-training objectives affect zero- and few-shot performance §3.2.

Method

Architecture §4.1, app. A. Each image is turned into 36 region features, produced by a detector trained on Visual Genome. The region features are appended to the text tokens in the encoder §4.1. The decoder generates target text. The base model follows the VL-T5 layout and the large model the T5 layout app. A, tab. 9.

Training objective §4.1, eq. 1. Training minimises the negative log-likelihood of the target tokens, given the input text and the image:

L_\theta = -\sum_{i=1}^{|y|} \log P_\theta(y_i \mid y_{<i}, x, v) eq. 1

Here x is the input text, v the image regions and y the target tokens. The model is not task-specific, so no task heads are added §4.1.

Pre-training §4.2, fig. 3.

  1. Data: MS COCO and Visual Genome image-caption data, 9.18M image-text pairs over 180K distinct images §4.2.
  2. Prefix language modeling (PrefixLM) follows T5. A caption is cut at a random point. The part before the cut goes to the encoder with the image, and the rest is the decoder target §4.2.
  3. Masked language modeling (MaskedLM) follows VL-T5. About 15% of the input words are replaced by numbered sentinel tokens, and the decoder generates the hidden spans §4.2.

Low-resource adaptation §5, fig. 2. A prompt template P turns an input and a label into input text x and target text y. The model is trained on these with the same loss eq. 1. At inference the same prompt is used, the model generates text, and the target template is removed to leave the label §5.

Prompt designs §5.1, tab. 1, tab. 11.

IDTaskInput text (paraphrased)
P1VQAthe question, then one sentinel
P2VQAa "question: ... answer:" frame around the question
P3VQAthe P2 frame, followed by one sentinel
Q1 / Q2 / Q3Captioninga fixed phrase meaning "a picture of", "a photo of" or "an image of"
noisy (ablation)VQA5 irrelevant instructions, 5 strings of random vocabulary tokens, 5 unrelated captions tab. 11
miniImageNet5-way classificationa fixed phrase ending in one sentinel

VQA targets are either the answer alone or a sentinel followed by the answer. In the main experiments the target is the sentinel plus answer, and the input is P3 §6.1. For captioning, the target is the caption itself tab. 1. VQA is treated as generation, so answers are produced as strings, not chosen from a fixed answer list §5.1.

Pseudocode: pre-training examples §4.2. Each caption gives two examples, one per objective. Captions of two words or fewer are skipped for T5 backbones, and duplicate sentences are removed pretrain_data.py.

Pseudocode
# Build two pre-training examples from one caption  [§4.2]
# Captions of two words or fewer are skipped (code)  [code: src/pretrain_data.py#L72]
def make_examples(caption, image_regions):
    words = caption.split()
    # PrefixLM: cut at one random word position  [§4.2] [code: src/preprocess.py#L104]
    cut = random_int(1, len(words) - 1)
    prefix_example = {
        "encoder_text": words[:cut],
        "target_text": words[cut:],
        "image": image_regions,
    }
    # MaskedLM: hide about 15% of words, at least one, in numbered spans
    # [§4.2] [code: src/preprocess.py#L30]
    masked_example = mask_spans(words, mask_ratio=0.15)
    masked_example["image"] = image_regions
    return [prefix_example, masked_example]

Pseudocode: loss for one example eq. 1 pretrain_model.py.

Pseudocode
# Loss for one example, following eq. 1  [eq. 1]
# The code divides the summed token loss by the number of target tokens
# [code: src/pretrain_model.py#L41]; eq. 1 has no such division
def example_loss(model, encoder_text, image, target_tokens):
    # Negative log-likelihood of each target token  [eq. 1]
    token_nll = model.token_nll(encoder_text, image, target_tokens)
    # Average over the target tokens of this example  [code: src/pretrain_model.py#L41]
    return token_nll.sum() / len(target_tokens)

Pseudocode: few-shot split and VQA input §3.1, §5.1.

Pseudocode
# Few-shot train and dev sets from one shuffled pool  [§3.1]
# Code: seeded shuffle, disjoint 16-example slices  [code: src/vqa_data.py#L95-L101]
def few_shot_split(pool, data_seed, num_data=16):
    shuffled = seeded_shuffle(pool, data_seed)
    train = shuffled[:num_data]
    dev = shuffled[num_data:2 * num_data]
    return train, dev
Pseudocode
# VQA input text with prompt P3  [§5.1] [code: src/vqa_data.py#L185]
# The whole input is cut to 20 tokens before image tokens are added
def vqa_input_text(question):
    return "question: " + question + " answer: <extra_id_0>"
Pseudocode
# VQA target text: sentinel, then answer  [§6.1] [code: src/vqa_data.py#L231]
# Cut to 10 tokens  [code: src/vqa_data.py#L231]
def vqa_target_text(answer):
    return "<extra_id_0> " + answer

Hyperparameters

Values marked "not stated" do not appear in the paper. Code values come from the official repository at code_commit.

NameValueWhereNotes
Backbone, base12+12 layers, width 768, FFN 3,072, 12 heads of size 64; 224M paramsapp. A tab. 9 tab. 2VL-T5 layout
Backbone, large24+24 layers, width 1,024, FFN 4,096, 16 heads of size 64; 740M paramsapp. A tab. 9 tab. 2T5 layout; the repo's scripts set only t5-base pretrain.sh
Image input36 region features per image§4.1 param.pyFeature dimension 2048 in code param.py; not stated in paper
Pre-training data9.18M pairs, 180K images (MS COCO, Visual Genome)§4.2Code trains on the COCO re-split train set and a Visual Genome set named vgnococo pretrain.sh
Pre-training batch1,280 (base), 800 (large)§6.1Code: 320 per process pretrain.sh; see Paper vs code
Pre-training epochs30§6.1pretrain.sh
Pre-training learning rate1e-4§6.1pretrain.sh
Pre-training warm-up5% of steps, linear§6.1pretrain.sh; linear decay in trainer_base.py (decay not stated in paper)
Pre-training optimisernot statedpretrain.shAdamW; eps 1e-6, betas 0.9 and 0.999, weight decay 0.01 param.py param.py
Pre-training gradient clipnot statedpretrain.sh1.0 in code
Pre-training precisionnot statedpretrain.shMixed precision (fp16) in code
Dropoutnot statedparam.py0.1 default in code
MaskedLM mask rate15% of input words§4.2At least one word; preprocess.py
Input text lengthnot statedvqa_data.py20 tokens for VQA input; 20 for pre-training text param.py
Few-shot train and dev size16 and 16§3.1 §6.1Disjoint slices of one shuffled split vqa_data.py
Few-shot epochs200§6.1 §3.4VQA.sh
Few-shot learning rate5e-5§6.1VQA.sh
Few-shot warm-up5%§6.110% in code VQA.sh
Few-shot batch sizenot stated30 in code VQA.sh
Few-shot gradient clipnot stated5 in code VQA.sh
Few-shot optimisernot statedAdamW in code VQA.sh
Checkpoint selectionbest on dev set§3.4VQA: highest dev accuracy vqa.py
Decodingnot statedBeam search with 5 beams, up to 20 generated tokens in code VQA.sh param.py
Prompt, VQAP3, target sentinel plus answer§6.1Code default is no prompt param.py; the flag --prompt 3 is needed VQA.sh
Prompt, captioning"an image of"§6.1Code needs --prefix image flickr.sh
Seeds and splitsfive random train/dev splits, averaged§3.4Code runs one data seed per run; default 9595 param.py; scripts use 42 VQA.sh

Results

Author-reported, on the paper's own splits and evaluation sets §3.4. Model sizes are given as in the paper. Few-shot results are averages over five splits.

Zero-shot VQA tab. 2: accuracy.

ModelSizeVQAv2OK-VQAGQA
Unified VLP122M0.0--
VL-T5 (no VQA pre-training)224M13.55.86.3
Frozen7B29.55.9-
PICa175B-17.5-
FewVLM base224M43.411.627.0
FewVLM large740M47.716.529.3

Few-shot VQA tab. 3: accuracy, 16 examples unless noted.

ModelSizeVQAv2OK-VQAGQA
Unified VLP122M24.3--
VL-T5 (no VQA pre-training)224M31.812.719.6
Frozen (4 examples)7B38.212.6-
PICa (16 examples)175B54.343.3-
FewVLM base, 4 examples224M45.114.526.9
FewVLM base224M48.215.032.2
FewVLM large740M51.123.135.7
Fully fine-tuned (reference)-72.6-61.5

Zero-shot captioning tab. 4: CIDEr and SPICE.

ModelSizeNoCaps CIDErNoCaps SPICEFlickr30k CIDErFlickr30k SPICE
Unified VLP122M--24.97.2
VL-T5 (no VQA pre-training)224M4.45.32.62.0
SimVLM (huge)-101.4---
FewVLM base224M42.28.531.010.0
FewVLM large740M47.79.136.510.7

Few-shot captioning tab. 5: CIDEr and SPICE, 16 examples.

ModelSizeNoCaps CIDErNoCaps SPICEFlickr30k CIDErFlickr30k SPICE
Unified VLP122M--28.89.4
VL-T5 (no VQA pre-training)224M22.06.812.88.3
FewVLM base224M48.610.032.612.8
FewVLM large740M53.110.437.013.5
Fully fine-tuned (reference)-112.213.167.417.0

5-way miniImageNet tab. 6: accuracy by shots per class.

Model1 shot3 shots5 shots
Frozen (7B)14.534.733.8
FewVLM base, no prompt48.075.082.6
FewVLM base57.078.084.2
FewVLM large57.178.384.4
AFHN62.3-78.1

Prompt study, FewVLM base, zero-shot tab. 7: VQAv2 accuracy and Flickr30k CIDEr.

PromptVQAv2Flickr30k
No prompt3.79.6
P1 / Q1 ("a picture of")9.915.2
P2 / Q2 ("a photo of")19.025.6
P3 / Q3 ("an image of")43.431.0

Pre-training objectives, FewVLM base tab. 8: VQAv2, GQA and Flickr30k CIDEr.

ObjectiveZero-shot VQAv2Zero-shot GQAZero-shot Flickr30kFew-shot VQAv2Few-shot GQAFew-shot Flickr30k
MaskedLM42.425.14.646.031.418.5
PrefixLM11.96.726.840.827.631.8
MaskedLM + PrefixLM43.427.031.048.232.232.6

Appendix app. B, app. D, tab. 10, tab. 12. COCO captioning, zero-shot and few-shot CIDEr/SPICE: FewVLM base 84.5/16.2 and 98.7/18.9; FewVLM large 92.1/17.3 and 100.4/19.1; SimVLM (huge) 102.3/22.1 zero-shot tab. 10. Pre-training on Conceptual Captions instead of COCO and Visual Genome lowers few-shot VQAv2 from 48.2 to 36.7, GQA from 32.2 to 25.9 and Flickr30k CIDEr from 32.6 to 22.3 tab. 12.

Paper vs code

Checked against the official repository at code_commit 3c17bc980a0d28cfb06eecdcf2e13a991c09a00e. The repository has no LICENSE file, so no code is quoted here; only paths and line numbers are cited.

  • Few-shot warm-up. Paper: 5% §6.1. Code: 0.1 in all five few-shot scripts VQA.sh flickr.sh COCOCaption.sh Nocaps.sh OKVQA.sh.
  • Pre-training batch. Paper: 1,280 for base and 800 for large §6.1. Code: 320 per process with a per-process data loader pretrain.sh pretrain_data.py, so the total batch is 320 times the number of GPUs. The README example uses 8 GPUs, which would give 2,560 README.md. The GPU count behind the paper's 1,280 is not stated.
  • Pre-training loss. Paper: eq. 1 sums the token losses eq. 1. Code: sums per example, then divides by the number of target tokens, then averages over the batch pretrain_model.py pretrain_model.py. The two differ in scale for examples of different lengths.
  • Examples per caption. The paper describes two objectives §4.2. Code builds both examples from every caption, so each caption appears twice per epoch pretrain_data.py.
  • Pre-training checkpoint. The paper does not say which checkpoint is used downstream. Code saves a checkpoint every epoch; the best-loss copy is commented out pretrain.py. The VQA script loads the last one, Epoch30 VQA.sh.
  • Default prompt. Paper: P3 for VQA and "an image of" for captioning §6.1. Code: the default is no prompt param.py and captioning uses an empty prefix when none is given flickr_data.py. The scripts' example commands pass --prompt 3 and --prefix image, but the scripts themselves do not VQA.sh flickr.sh.
  • Sentinel token. The paper writes <text_1> §5.1. Code uses T5's <extra_id_0> vqa_data.py preprocess.py. This is a naming difference; behaviour should match, but it is not verified.
  • Few-shot seeds. Paper: average over five random splits §3.4. Code: one data seed per run, and the example commands use seed 42 VQA.sh. No script runs the five-split average.
  • Decoding. The paper does not state a decoding method. Code uses beam search with 5 beams VQA.sh.
  • Large model. The paper uses a T5 layout for the large model app. A. The scripts set only t5-base pretrain.sh, so no large-model configuration is in the repository.
  • Not in the repository. The Faster R-CNN feature extractor, the pre-training image-caption data, and any miniImageNet pipeline are not in the repository. Features are read from precomputed HDF5 files vqa_data.py. The README points to VL-T5 for feature extraction and to a download folder for the data README.md, "Datasets".
  • Training target for VQA. Not stated in the paper. Code draws the answer for each example at random, weighted by the answer scores, each time the example is loaded vqa_data.py.

No discrepancies were found for the learning rates (1e-4 for pre-training, 5e-5 for few-shot), the pre-training epochs (30), the few-shot epochs (200), or the 15% masking rate.

Pitfalls

  • Set the prompt explicitly. The code default is no prompt param.py. Without a prompt, zero-shot VQAv2 is 3.7 against 43.4 with P3 tab. 7. Few-shot VQA and captioning need the same flag VQA.sh.
  • Zero-shot is very sensitive to wording. On Flickr30k the three captioning phrases give CIDEr from 15.2 to 31.0 tab. 7. With noisy prompts, zero-shot accuracy is close to 0 §6.5.2. Few-shot results move much less §6.5.2, fig. 4.
  • Truncation cuts the template. The whole prompt-plus-question string is cut to 20 tokens before image tokens are added vqa_data.py modeling_t5.py. For long questions, the trailing "answer" frame and the sentinel can be lost. The paper does not mention this.
  • Answer targets are sampled. Each time a VQA example is loaded, its answer is drawn from the weighted answer list, so the target can change between epochs vqa_data.py. The paper does not state this.
  • Few-shot variance is large. The paper averages five splits §3.4. The target prompt "[A]" alone has a larger spread across training sizes than the sentinel version §6.5.2, fig. 6. A single run from the scripts is one sample.
  • Checkpoint choice. For VQA the best dev accuracy wins, and ties go to the later epoch vqa.py. Pre-training keeps every epoch and the scripts use the last one pretrain.py.
  • Masking produces one sentinel per span. Adjacent masked words merge into one span, and the number of masked words is at least one preprocess.py preprocess.py. The number of sentinels is therefore at most the number of masked words.
  • Prefix split is uniform over word positions. The cut is drawn uniformly from 1 to n−1 words preprocess.py, so a one-word caption cannot be split. The code skips captions of two words or fewer for T5 backbones pretrain_data.py.
  • Abstract margin. The abstract's 18.2-point margin over Frozen matches the large model's zero-shot VQAv2 (47.7) minus Frozen (29.5) tab. 2. The base model's margin is 13.9 (derived from the same table). The text's "31 times" refers to model size §6.2.
  • NoCaps has no training split in the paper. It uses COCO captioning training data §3.4; the script does the same Nocaps.sh.
  • Cost. The paper reports no inference cost or latency. The code decodes with 5 beams VQA.sh, so reported speed does not transfer to greedy decoding.
  • Version. Compiled from arXiv v2 (updated 2022-03-15). The licence was recorded by wikillm fetch from the arXiv metadata. Other versions may differ.

Lineage

  • Builds on: VL-T5 for the base layout and MaskedLM app. A §4.2; T5 for the large layout and the PrefixLM objective app. A §4.2; prompt-based few-shot work in NLP §1, §3.2.
  • Compared against: Frozen, PICa, SimVLM, Unified VLP, VL-T5 without VQA pre-training, AFHN, and fully fine-tuned reference models §3.5 tab. 2–6.
  • Superseded by: none known from content in this wiki.

Source and license

Adapted from “A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models” by Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren (arXiv:2110.08484), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/woojeongjin/FewVLM at commit 3c17bc980a0d (none (no LICENSE file in repository)). Any quoted code is shown under that license.