InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

In plain words

Vision-language models look at a picture and answer in text. Many of them are built for one job, such as captioning or answering questions about a photo. This paper asks how to get one model that follows whatever the user asks about an image, the way chat models follow typed instructions.

The approach starts from an existing model, BLIP-2, which pairs a frozen image encoder and a frozen language model through a small trainable bridge called a Query Transformer (Q-Former). The authors gather many image datasets, rewrite each one as an instruction with an answer, and train only the bridge on them. A useful twist is that the instruction is shown to the bridge as well as to the language model. The bridge can then pull out the image details that matter for that particular question, not the same summary every time.

The result is a model that handles tasks and datasets it was not trained on better than its starting point and a larger earlier model, Flamingo-80B. It matters because it shows that instruction tuning, which works for text-only models, also carries over to images, and that a small amount of training on a bridge can be enough.

TL;DR

InstructBLIP instruction-tunes BLIP-2 on 13 held-in vision-language datasets, rewritten as instructions from 26 datasets in total. The image encoder and the LLM stay frozen; only the Q-Former is trained, and it also receives the instruction tokens. Datasets are sampled with probability proportional to the square root of their size. The authors report new zero-shot results on all 13 held-out datasets and a fine-tuned ScienceQA score of 90.7%. Use it when you need a reference recipe for instruction-aware visual features on a frozen LLM, but note that the official repository lacks the instruction-tuning training configuration (see Paper vs code).

Problem

Instruction tuning works well for text-only LLMs, but vision-language tasks are more varied because the image input comes from many domains §1. Multitask training with a shared input format, without instructions, does not generalize to unseen datasets and tasks in the authors' experiments §1 §3.4. Models that only add a visual front end and train it on image captions cover visual descriptions well but not tasks that need more than that §1.

Method

The model is BLIP-2 with a frozen ViT-g/14 image encoder, a Q-Former, and a frozen LLM. A linear projection maps the Q-Former outputs into the LLM input space §2.3 fig. 2. Training changes only the Q-Former.

Data and instructions:

1. Collect 26 public datasets across 11 task types, and convert each into an instruction, answer pair §2.1 app. C. Each task has 10 to 15 hand-written templates §2.1 app. D. For datasets whose answers are short, some templates say "short" or "briefly" to avoid teaching the model to always answer briefly §2.1. LLaVA-Instruct-150K is used without extra templates §2.1.
2. Split the data: 13 datasets are held in for training, and 13 are held out for zero-shot tests §2.2 app. C. Four task categories are withheld entirely (visual reasoning, video QA, visual conversational QA, image classification) §2.2. The split is designed so no evaluation data appears in training §2.2.
3. For datasets with scene text, append OCR tokens to the instruction §2.2 app. D.

Instruction-aware visual features:

4. The Q-Former takes K learnable query embeddings. The instruction tokens go into its self-attention with the queries, and the queries cross-attend to the frozen ViT features §2.3 blip2_t5_instruct.py.
5. The K output vectors go through the linear projection and are fed to the frozen LLM as soft prompts, followed by the instruction tokens for the LLM §2.3 blip2_t5_instruct.py.
6. Training uses the standard language-modeling loss on the response §2.2 blip2_t5_instruct.py.

Dataset balancing:

7. Sample a training example from dataset d with probability proportional to the square root of its size, then adjust some dataset weights by hand. The manual adjustment lowers A-OKVQA (multiple-choice) and raises OKVQA (open-ended) §2.4. The paper's formula is written inline in the text, not as a numbered equation.

p_d = \frac{\sqrt{S_d}}{\sum_{i=1}^{D} \sqrt{S_i}} §2.4

Inference:

8. Open-ended tasks (captioning, open VQA) use free generation §2.5.
9. Classification and multiple-choice tasks use vocabulary ranking: the model generates, but only over a list of candidates, and the candidate with the highest log-likelihood wins. Ranking applies to the science and icon QA sets, the multiple-choice version of A-OKVQA, the meme, dialog and both video QA benchmarks §2.5. Binary tasks expand the labels into a broader set of words, such as "yes" and "true" §2.5.
10. Video QA uses four uniformly sampled frames. Each frame goes through the image encoder and Q-Former separately, and the features are concatenated before the LLM §2.5. Concatenation means the LLM receives four times as many visual vectors (derived from §2.5).

# Instruction-aware training step  [§2.2] [§2.3]
# Input:  image (B, 3, 224, 224); instruction, response: lists of text
# Output: scalar language-modeling loss

def training_step(image, instruction, response):
    # Encode the image with the frozen ViT  [§2.3] [§2.6]
    image_embeds = frozen_vit(image)                 # (B, N, d_v)

    # Q-Former: learned queries read the instruction and the image  [§2.3]
    query_outputs = qformer(
        queries,
        instruction_tokens,
        image_embeds,
    )                                                # (B, K, d_q)

    # Project the query outputs into the LLM embedding space  [§2.3]
    soft_prompts = linear_projection(query_outputs)  # (B, K, d_llm)

    # The frozen LLM reads soft prompts and the instruction  [§2.3]
    # Loss is the language-modeling loss on the response  [§2.2]
    return frozen_llm_loss(soft_prompts, instruction, response)

# Balanced dataset sampling  [§2.4]
# Input:  sizes, the number of training samples per held-in dataset
# Output: sampling probability per dataset

def sampling_probabilities(sizes):
    # Square root of each dataset size  [§2.4]
    roots = [sqrt(size) for size in sizes]
    # Normalize so the probabilities sum to one  [§2.4]
    total = sum(roots)
    # Manual per-dataset weight changes are not stated  [§2.4]
    return [root / total for root in roots]

Hyperparameters

Values are from the paper unless the "Where" column says code. Items marked "not stated" were not found in the paper; code is cited where it gives a value.

Results

Author-reported. Setup: 13 held-out datasets, zero-shot, with the prompts in app. E. Baselines are BLIP-2 (four variants) and Flamingo (3B, 9B, 80B). The metrics are CIDEr for NoCaps and Flickr30K, MRR for Visual Dialog, AUC for HatefulMemes, and top-1 accuracy for the rest tab. 1. The paper does not say whether baseline rows are re-run or copied from earlier papers.

Zero-shot, held-out datasets tab. 1:

Headline claims in the text §3.1:

• New zero-shot results on all 13 held-out datasets §3.1 tab. 1.
• FlanT5-XL improves on BLIP-2 FlanT5-XL by an average relative 15.0% §3.1.
• FlanT5-XL outperforms Flamingo-80B on all six shared datasets, with an average relative gain of 24.8% §3.1.
• Up to 47.1% relative gain on MSRVTT-QA over the previous best §3.1.

Ablation on FlanT5-XL and Vicuna-7B tab. 2. Held-in average is over COCO Caption, OKVQA, A-OKVQA and TextCaps tab. 2 caption. Metrics differ by column; the paper does not give a per-cell metric list.

The text reads these rows as: removing instruction-aware features hurts everywhere, most on ScienceQA and iVQA; removing balancing makes training uneven across datasets and lowers the overall score §3.1 tab. 2.

Finetuning on downstream datasets tab. 3 §3.5. The paper reports these as new state of the art on ScienceQA (IMG), OCR-VQA and A-OKVQA, and OKVQA is higher for PaLM-E (562B) §3.5.

DA = direct answer, MC = multiple choice tab. 3.

Other results: multitask learning without instructions does about as well as instruction tuning on held-in data, but much worse on unseen data, where it matches BLIP-2 §3.4 fig. 3. Visual Dialog is scored with MRR rather than NDCG, which the authors argue fits zero-shot evaluation better §3.1.

Paper vs code

Code: salesforce/LAVIS at commit 05cb670f47296522e24e7727e366b4a997d28582, the repository the paper links in its abstract. The code's InstructBLIP model classes are in lavis/models/blip2_models/ and the configs are in lavis/configs/models/blip2/ and lavis/projects/instructblip/.

Matches:

• The instruction goes to the Q-Former as well as the LLM. The default is on for both FlanT5 and Vicuna variants §2.3 blip2_t5_instruct.py blip2_t5_instruct.py.
• ViT and LLM are frozen §2.2 blip2_t5_instruct.py blip2_t5_instruct.py.
• Loss is the language-modeling loss on the response. Input padding in the targets is masked with -100 §2.2 blip2_t5_instruct.py.
• ViT frozen with 224-pixel images in the configs §2.6 §3.5 blip2_instruct_vicuna7b.yaml.

Differences and gaps:

• The instruction-tuning training configuration is not in the repository. The lavis/projects/instructblip/ folder holds evaluation and downstream configs only. The batch sizes, 60K steps, learning-rate schedule and dataset sampling weights therefore cannot be checked in code. The train_dataset_ratios entries in the repository belong to XInstructBLIP, a different paper image_train.yaml.
• The downstream configs use init LR 1e-5, warmup from 1e-8 over 1,000 steps, weight decay 0.05, 1 epoch and batch size 16 caption_coco_flant5xxl_eval_val.yaml. The paper does not state finetuning hyperparameters §3.5, so these are not confirmed as the paper's finetuning setup.
• The code has a few-shot embedding path that the paper does not describe blip2_t5_instruct.py. Whether the released models used it is not verifiable from this repository.
• Balanced sampling (√size with manual weights) has no implementation in the InstructBLIP code §2.4.
• The paper gives no max-token limits; the code sets 128 input and 256 output tokens blip2_t5_instruct.py.

Pitfalls

• Model size conflict. §2.6 gives FlanT5-XL as 3B parameters; §3.1 calls the same model 4B. The paper does not resolve this. Do not rely on either number for memory planning without checking the checkpoint.
• Sampling weights are not reproducible from the paper. The formula is given, but the manual adjustments have no values, and there is no sampling code for this paper in the repository §2.4.
• Checkpoint selection. A single checkpoint per model is used for all datasets, but the selection metric is not stated §2.6.
• Vicuna variants need Q-Former pretraining. BLIP-2 has no Vicuna checkpoints, so the authors repeated BLIP-2's pretraining with Vicuna §2.6. The data and steps for that run are not stated in this paper.
• Truncation drops the start of long text. The text tokenizers truncate from the left, so for inputs over 128 tokens, the earliest tokens are lost blip2_t5_instruct.py blip2_t5_instruct.py. This affects long OCR lists and dialog histories. The 128-token limit is a code default, not a paper value.
• Mixed metrics in the ablation. The held-in average combines captioning and accuracy scores. The paper does not say how the average is weighted tab. 2 caption.
• Video doubles the visual tokens. Concatenating four frames' Q-Former outputs gives four times the visual vectors (derived from §2.5). The paper does not discuss the extra cost.
• Task-specific decoding. Classification and multiple-choice use candidate ranking, and the rest use generation §2.5. Results from the two modes are not directly comparable. Decoding settings for evaluation are not stated in the paper.
• Held-out prompts matter. Zero-shot prompts are listed in app. E. The repository's eval configs use slightly different strings in places, for example "Short Answer:" in qa_okvqa_flant5xl_eval.yaml versus "Short answer:" in app. E. Use the exact prompt you are reproducing.
• Baselines. The paper does not state whether BLIP-2 and Flamingo numbers were re-run or copied tab. 1.
• Qualitative comparisons. Appendix comparisons with GPT-4, LLaVA and MiniGPT-4 use responses taken from other papers or from official demos app. B. They are not controlled comparisons.
• Training data licenses. The 26 datasets have their own terms, including dataset-specific restrictions. This card does not license any of them app. C.
• Figure and table numbers are inferred. Numbers such as tab. 1 and fig. 2 come from the order of the LaTeX source, not from the compiled PDF. Check them against the PDF before you rely on them.

Lineage

• Builds on: BLIP-2, with its Q-Former and frozen-LLM setup §1 §2.3. The source bibliography gives no arXiv id for it, and there is no card for it in this wiki. FlanT5 §2.6 and Vicuna (a LLaMA instruction-tuned model) §2.6 are the LLMs used.
• Compared against: Flamingo-3B/9B/80B and BLIP-2 variants tab. 1; multitask learning without instructions §3.4 fig. 3.
• Qualitatively compared with: GPT-4, LLaVA and MiniGPT-4 app. B.
• Superseded by: not known from content in this wiki.