BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

In plain words

Large language models can write fluently, but they have never seen pictures, and training a vision model and a language model together from scratch is expensive. This paper asks whether you can reuse both pre-trained parts as they are and only train a small piece in between.

That small piece is called a Querying Transformer (Q-Former). It holds a fixed set of learnable query slots. The slots look at the image features and keep only the parts that matter for the text, so the language model receives a short, useful summary instead of the full image. Training happens in two steps. First the bridge learns to line up images with their captions. Then it learns to feed the summary to the language model so the model can write about the picture.

Because the two large models stay frozen, the trainable part is small and training is cheaper than end-to-end methods. The authors report that the result answers visual questions and writes captions well, and can follow typed instructions about an image without being trained for that.

TL;DR

BLIP-2 trains a 188M-parameter Q-Former between a frozen image encoder and a frozen LLM §3.1. Stage 1 aligns it with images and text using three losses (contrastive, matching, and generation) §3.2. Stage 2 connects its output to the LLM as soft visual prompts §3.3. Author-reported: 65.0% zero-shot VQAv2 test-dev, 8.7 points above Flamingo-80B tab. 1 tab. 2. Read the Pitfalls section before comparing parameter counts or reproducing the schedule.

Problem

End-to-end vision-language pre-training is expensive because it trains large models on large datasets §1. Freezing a strong LLM avoids that cost, but an LLM has not seen images during its own pre-training, so alignment is hard §1. The authors argue that the image-to-text generation loss used by earlier frozen methods (Frozen, Flamingo) is not enough to close this gap §1.

Method

BLIP-2 has two stages. Both keep the image encoder and the LLM frozen. The Q-Former is trained in both stages §3 §3.3. The stage-2 projection layer is also trained in the code, though the paper does not say so explicitly (see Paper vs code).

Model architecture §3.1 fig. 2

• The Q-Former has two transformer submodules that share self-attention layers: an image transformer, which reads the frozen image features, and a text transformer, which works as text encoder and text decoder §3.1.
• The image transformer takes a fixed set of learnable query embeddings. Queries attend to each other through self-attention and to the frozen image features through cross-attention, which is inserted every other transformer block. The queries can also attend to the text through the shared self-attention §3.1.
• The Q-Former is initialised from BERT-base. Its cross-attention layers start at random. The queries are model parameters §3.1.
• The paper uses 32 queries of dimension 768. The output Z is 32 x 768, against a frozen ViT-L/14 feature map of 257 x 1024, which is the bottleneck §3.1.
• Different pre-training objectives use different self-attention masks to control query-text interaction §3.1 fig. 2.

Stage 1: representation learning from a frozen image encoder §3.2

Three objectives share inputs and weights and differ only in their masks §3.2 fig. 2:

• Image-text contrastive (ITC). Each query output is compared with the text representation, which is the output at the first text token. The highest query score gives the image-text similarity. A unimodal mask stops queries and text from seeing each other. In-batch negatives are used §3.2.
• Image-grounded text generation (ITG). A multimodal causal mask lets the queries attend to each other but not to the text. Each text token attends to all queries and to earlier text tokens. A new [DEC] token replaces [CLS] as the first text token §3.2. The queries are therefore forced to hold everything needed to write the caption.
• Image-text matching (ITM). A bidirectional mask lets all queries and text attend to each other. Each query output goes into a two-class linear classifier and the logits are averaged over queries. Hard negatives are mined as in ALBEF and BLIP §3.2.

The three losses are optimised jointly §3.2.

Stage 2: vision-to-language generation from a frozen LLM §3.3 fig. 3

• A fully connected layer projects each Q-Former output into the LLM's text embedding width. The projected vectors are prepended to the text embeddings as soft visual prompts §3.3 fig. 3.
• Decoder-only LLM (OPT family): the language-modelling loss is applied to the text, conditioned on the visual prompts §3.3.
• Encoder-decoder LLM (FlanT5 family): the prefix of a text goes into the encoder together with the visual prompts. The suffix is the decoder target, trained with a prefix language-modelling loss §3.3.

The paper does not state whether the projection is trained. The code trains it, so see the Paper vs code section.

Pre-training data and frozen components §3.4 fig. 4

• Images: 129M in total, the same set as BLIP: COCO, Visual Genome, CC3M, CC12M, SBU, and 115M images from LAION400M §3.4.
• Synthetic captions (CapFilt, from BLIP): 10 captions per web image come from the BLIP-large captioner. They are ranked with the web caption by image-text similarity from a CLIP ViT-L/14 model. The top two are kept, and one is sampled at each step §3.4.
• Frozen image encoders: CLIP ViT-L/14 and EVA-CLIP ViT-g/14. The paper removes the last ViT layer and uses the second-to-last layer's output, which it says works slightly better §3.4.
• Frozen LLMs: OPT (decoder-only, unsupervised) and FlanT5 (encoder-decoder, instruction-tuned) §3.4.
• Image augmentation: random resized crop and horizontal flip, at 224 x 224 §3.4.

Pseudocode

The paper has no numbered equations or algorithm blocks. The pseudocode below is our condensed reading of §3.2 and §3.3. Shapes are shown in comments.

# Stage 1: one training step, three objectives  [§3.2] [fig. 2]
# Input:  images (B, 3, H, W); captions: list of B strings
# Output: scalar loss

def stage1_step(images, captions):
    # Frozen image encoder; its weights never change  [§3.4]
    image_embeds = frozen_vit(images)              # (B, P, D_vit)

    # 32 learnable queries, 768-d each  [§3.1]
    queries = learned_queries.expand(B, -1, -1)    # (B, 32, 768)

    # ITC: unimodal mask, so queries and text do not see each other  [§3.2]
    query_out = qformer_image_pass(queries, image_embeds)  # (B, 32, 768)
    text_cls = qformer_text_pass(captions)         # (B, 768), first token
    # Score each image by its best-matching query  [§3.2]
    # E: projection width, not stated in the paper
    sims = max_over_queries(
        normalize(project_img(query_out)) @ normalize(project_txt(text_cls)).T
    )                                              # (B, B)
    loss_itc = contrastive_loss(sims, temperature)  # in-batch negatives

    # ITG: causal mask; queries see only queries  [§3.2]
    # Text sees the queries and earlier text tokens  [§3.2]
    # The first text token is [DEC] instead of [CLS]  [§3.2]
    loss_itg = next_token_loss(queries, image_embeds, dec_tokens(captions))

    # ITM: bidirectional mask  [§3.2]
    # Hard negatives mined from the ITC similarities  [§3.2]
    negatives = mine_hard_negatives(sims)
    logits = itm_head(pair_outputs).mean(over_queries)  # (3B, 2)
    loss_itm = classification_loss(logits, match_labels)

    return loss_itc + loss_itm + loss_itg

# Stage 2: vision-to-language generation  [§3.3] [fig. 3]
# Input:  images; texts (decoder LLM) or prefix/suffix pairs (encoder-decoder)
# Output: scalar language-modelling loss

def stage2_step(images, texts):
    # Q-Former output for each image  [§3.1]
    query_out = qformer(learned_queries, frozen_vit(images))  # (B, 32, 768)
    # Project to the LLM's embedding width  [§3.3]
    # The paper's FC layer; trained in the code, see Paper vs code  [§3.3]
    soft_prompts = fc_projection(query_out)        # (B, 32, D_llm)

    if llm_is_decoder_only:
        # Visual prompts prepended to the text  [§3.3]
        inputs = concat(soft_prompts, embed(texts))
        return lm_loss(frozen_llm(inputs), texts)

    # Encoder-decoder: prefix to the encoder, suffix is the target  [§3.3]
    inputs = concat(soft_prompts, embed(prefixes))
    return lm_loss(frozen_llm(inputs, decoder_target=suffixes), suffixes)

Hyperparameters

Values marked "app." come from the appendix hyperparameter tables. Those tables have no numbers, so they are cited by position: first, second and third table.

Results

Author-reported. Setup: frozen encoders and LLMs, Q-Former trained. Compared against the published numbers of the listed baselines §4.

Zero-shot VQA, VQAv2 and OK-VQA tab. 2. Accuracy, in %.

The paper reports BLIP-2 as best on VQAv2 and GQA, and second to Flamingo-80B on OK-VQA §4.1 tab. 2.

Overview, zero-shot tab. 1. Single-model headline numbers.

Image captioning tab. 3. Fine-tuned on COCO, evaluated on the COCO Karpathy test split and on NoCaps validation.

VQA fine-tuned, VQAv2 open-ended generation tab. 4. Accuracy, in %.

Image-text retrieval tab. 5. R@1, in %. COCO is fine-tuned, Flickr30K is zero-shot.

Retrieval ablation tab. 6. COCO fine-tuning with and without the ITG loss, ViT-g: image to text R@1 84.5 (ITC+ITM) vs 85.4 (ITC+ITM+ITG); text to image R@1 67.2 vs 68.3 tab. 6.

Qualitative and other claims, not tabulated.

• Instructed zero-shot image-to-text generation with a text prompt after the visual prompt: examples only, no metrics §4.1 fig. 4.
• Without the first stage, zero-shot VQA is much lower for both LLM types, and OPT degrades as training proceeds (catastrophic forgetting). The figure gives the curves; no numeric values are in the text §4.1 fig. 5.
• In-context learning with few-shot VQA examples did not improve results in the authors' experiments §5.

Paper vs code

Official code: LAVIS, salesforce/LAVIS, at commit 05cb670f47296522e24e7727e366b4a997d28582. The paper links it in its author block. The code is licensed BSD-3-Clause LICENSE.txt. The BLIP-2 files are under lavis/models/blip2_models/, and the training configs under lavis/projects/blip2/. Code paths below are relative to the LAVIS root.

Differences found:

• Pre-training schedule and batch size. Paper: 250k stage-1 steps and 80k stage-2 steps, batch 2320 or 1680 (stage 1) and 1920 or 1520 (stage 2) §3.4. Code: 10 epochs per stage, batch 100 per GPU for stage 1 and 64 per GPU for stage 2, on 16 GPUs pretrain_stage1.yaml pretrain_stage2.yaml pretrain_stage1.sh. The configs do not give the paper's step counts, and the totals do not match the paper's batch sizes.
• Warmup and minimum learning rate. Paper: 2k warmup steps and a stage-2 minimum of 5e-5 §3.4. Code: stage 1 warmup 5000 steps pretrain_stage1.yaml; stage 2 warmup 2000 steps and minimum 1e-5 pretrain_stage2.yaml pretrain_stage2.yaml.
• Stage 2 initialisation. The stage-2 config loads the stage-1 checkpoint pretrain_stage2.yaml. The paper implies this but does not state it.
• AdamW beta2. Paper: 0.98 for pre-training §3.4. The optimiser uses beta2 from the run config, defaulting to 0.999 runner_base.py. The pre-training configs do not set it, so the default 0.999 applies. The retrieval fine-tune configuration for ViT-g uses 0.999 as in the appendix.
• ITC details not in the paper. Code: projection width 256, a learnable temperature initialised at 0.07, and cross-entropy with label smoothing 0.1, averaged over both directions blip2_qformer.py blip2_qformer.py blip2_qformer.py blip2_qformer.py.
• ITM negatives. Paper: hard negatives as in ALBEF and BLIP §3.2. Code: one negative image and one negative text per example are drawn by multinomial sampling from a softmax over the ITC similarities. Positives are masked out blip2_qformer.py. Sampling is not the same as picking the single hardest negative.
• Second-to-last ViT layer. Paper: remove the last layer and use the second-to-last output §3.4. In eva_vit.py, forward_features runs every block and returns the last block's output, and the final norm is commented out eva_vit.py. Q-Former then applies the layer norm it defines blip2_qformer.py. I did not find code that drops the last block in the files I read, so this is unresolved. A reimplementation should test both choices.
• Frozen parameters and precision. The ViT is frozen by default blip2_pretrain.yaml and loaded in FP16 blip2_pretrain.yaml. The OPT model is loaded in FP16 and its parameters set to requires_grad=False blip2_opt.py. The FlanT5 model is cast to BF16 blip2_t5.py. These match the paper's FP16/BF16 statement §3.4.
• Projection to the LLM. The OPT model adds a new linear layer, opt_proj, after the frozen LLM is set up blip2_opt.py. This answers the trainable-projection question in the paper's text.
• OPT caption targets. Each caption gets a newline appended before tokenising, and the loss is applied to the text tokens blip2_opt.py blip2_opt.py. The paper does not mention the newline.
• OPT zero-shot VQA prompt. Paper: "Question: {} Answer:" §4.1. The OPT VQAv2 evaluation config uses "Question: {} Short answer:" vqav2_zeroshot_opt_eval.yaml. The FlanT5 config matches the paper vqav2_zeroshot_flant5xl_eval.yaml.
• Length penalty. The VQA answer path defaults to -1, as the paper states blip2_t5.py. The general generate default is 1.0 blip2_t5.py.
• Consistent with the paper. Captioning prompt "a photo of" blip2_caption_opt2.7b.yaml; retrieval candidates k = 128 ret_coco_eval.yaml; retrieval batch 14 per GPU on 16 GPUs = 224 retrieval_coco_ft.yaml train_retrieval_coco.sh; captioning batch 16 per GPU on 16 GPUs = 256 caption_coco_ft.yaml.
• Not checked. The 129M pre-training mix and the CapFilt step. The stage-1 and stage-2 configs list coco_caption and vg_caption datasets, but I did not read the data-builder configs for the other sources.

Pitfalls

• The "8.7%" is a difference of percentage points. 65.0 minus 56.3 is 8.7 §4.1 tab. 2. It is not a relative gain.
• "54x fewer trainable parameters" depends on the count used. The abstract and §4.1 compare 188M with Flamingo-80B's 10.2B, which is about 54x §1 tab. 1. The same paper's Table 2 lists 108M for the XXL model, about 94x.
• Parameter counts disagree across tables. The Q-Former is 188M in the text and Table 1 §3.1 tab. 1, and 103M to 108M in Table 2 tab. 2. Tables 3 to 5 list 1.1B to 1.2B because they also train the image encoder tab. 3 tab. 4 tab. 5. BLIP is listed as 583M in Table 1 and 446M in Table 3. Count what is trainable in your own setup.
• The beta1 and beta2 values have a typo. The text writes beta1 = 0.9 and beta1 = 0.98 §3.4. The second value is beta2, following the appendix. The pre-training code uses 0.999 unless configured otherwise (see Paper vs code).
• The stage-2 minimum learning rate and warmup differ from the code. See Paper vs code. Reproducing the paper's numbers needs the paper's batch sizes and step counts, which the public configs do not provide.
• Retrieval cost is per candidate. Retrieval scores the ITC top 128 by running ITM on each pair §4.4. Cost per query is therefore 128 extra ITM forward passes. The paper does not report ITC-only retrieval numbers.
• "Zero-shot" captioning on NoCaps is after COCO fine-tuning. The model is fine-tuned on COCO and evaluated on NoCaps without NoCaps training data §4.2 tab. 3.
• Knowledge-heavy tasks depend on the LLM. The authors attribute the weaker OK-VQA result to the 70B-parameter Chinchilla LLM in Flamingo-80B having more knowledge than FlanT5-XXL §4.1.
• Hallucination and stale content. The authors list wrong knowledge, wrong reasoning paths, and outdated information about new image content as causes of bad outputs. They also say the model inherits LLM risks such as offensive output and leaked private information §5.
• Stage 1 matters for the generative model. Without stage 1, zero-shot VQA is much lower for both LLM types §4.1 fig. 5. Do not skip the first stage to save time.
• Compute is quoted on A100 40 GB hardware. The paper's "under 6 days" and "under 3 days" are for ViT-g with FlanT5-XXL on 16 GPUs §3.4. Scale them for other hardware.

Lineage

• Builds on: BLIP, whose pre-training data and CapFilt synthetic captions BLIP-2 reuses, and whose ITC, ITM and ITG objectives it adapts §1 §3.2 §3.4.
• Builds on the frozen-LLM approaches Frozen and Flamingo, which BLIP-2 argues are not enough alone §1 §4.1.
• Compared against: Flamingo, OFA, SimVLM, BEIT-3, CoCa, ALBEF, VinVL, OSCAR tab. 1 tab. 2 tab. 3 tab. 4 tab. 5.
• Extended by: InstructBLIP (see /papers/2305.06500), which instruction-tunes a BLIP-2 style model.