MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

In plain words

Models that can look at a picture and talk about it are hard to build well. GPT-4 showed striking skills, such as describing an image in detail or writing a website from a sketch, but how it does this was never published. The authors guess that the strength comes from the large language model behind it.

To test this, they take two existing parts that are already good at their own job: an image encoder and a strong chat-style language model. Both stay frozen, meaning their knowledge is not changed. They join the two with one small layer that the training does adjust. That layer turns image features into something the language model reads like words.

Training has two steps. The first teaches the layer from many image and caption pairs, which gives the model knowledge about pictures but makes its text broken or repetitive. The second step fine-tunes it on a small set of carefully written, detailed descriptions, which makes the answers read naturally. The result shows many of the same skills GPT-4 showed, such as explaining a meme or writing a poem about a picture, from training a single layer on a few GPUs for about ten hours.

TL;DR

MiniGPT-4 links a frozen BLIP-2 vision encoder and Q-Former to a frozen Vicuna LLM with one trainable linear layer. Stage 1 trains that layer on about 5M image-caption pairs. Stage 2 fine-tunes on about 3,500 curated detailed descriptions, which removes most repetition and fragmentation (author-reported). Use it as a reference for cheap LLM-based vision alignment, but read "Paper vs code" first: the released configs differ from the paper in several training details, including the stage 2 step count.

Problem

GPT-4 can describe images in detail, explain humour and build websites from sketches, but its technical details are not disclosed. The authors hypothesise that these abilities come from a more advanced LLM §1. In their comparison, BLIP-2 with the weaker Flan-T5 XXL language model did not show the same advanced abilities §4.2.

Aligning image features with a chat-style LLM on caption data alone gives outputs that are unnatural, with repeated words, fragments or irrelevant content §3.1. Earlier vision-language models with a weaker language model did not show the advanced skills §1 §4.2. The authors attribute the unnatural output to a style mismatch: caption datasets differ from how modern LLMs write §5.

Method

Architecture §3 fig. 1. Four parts:

1. Vision encoder: the EVA ViT used by BLIP-2 (described as ViT-G/14 §1), frozen §1 minigpt4_vicuna0.yaml.
2. Q-Former from BLIP-2, frozen §1 §3. It turns patch features into a fixed set of query outputs minigpt4.py.
3. One linear projection that maps those outputs into the LLM's embedding space. It is the only part trained in stage 1 §3.1 §1.
4. Vicuna as the language decoder, built on LLaMA, frozen §3 §1.

Projected image features are used as a soft prompt, placed in front of the LLM input, and the LLM is trained to generate the ground-truth text after them §3.1.

Stage 1: alignment pretraining §3.1.

1. Use a combined pool of image-caption pairs from Conceptual Captions, SBU and LAION, about 5M pairs in total §3.1.
2. Feed the projected image tokens with no text prompt. Only the image tokens come before the caption minigpt_base.py.
3. Append an end symbol to each caption and regress the caption tokens. The paper does not name the end symbol; the default in code is a newline minigpt_base.py [code: minigpt4/models/minigpt_base.py#L31].
4. Train for 20,000 steps with batch size 256 on 4 A100 (80GB) GPUs, about ten hours §3.1 §1.

Data curation for stage 2 §3.2. The authors build a detailed-description set with the stage 1 model:

1. Randomly pick 5,000 Conceptual Captions images §3.2.
2. Ask the stage 1 model for a detailed description, using Vicuna's chat format §3.2.
3. If the answer is 80 tokens or fewer, send a follow-up prompt that asks the model to continue, then join the two answers §3.2.
4. Ask ChatGPT to repair the text: remove repeats, meaningless characters and non-English sentences, and finish broken sentences. The exact instruction is in §3.2. The ChatGPT version is not stated.
5. Hard-coded filters drop apology phrases such as "I'm sorry", then the authors check each description by hand and remove leftover repetition. About 3,500 of the 5,000 pairs remain §3.2.

Stage 2: fine-tuning §3.3. Prompt format: a human turn with the image tokens and an instruction, then the assistant turn. The instruction is sampled from a predefined set of variants such as "describe this image in detail" §3.3 alignment.txt. The loss is not computed on the image and instruction part, only on the answer §3.3. The paper reports 400 steps with batch size 12, about seven minutes on one A100 §3.3.

Shared training step as implemented in the official code (pseudocode, our notation):

# One training step, used by both stages  [§3.1] [§3.3]
# Input:  image (B, 3, 224, 224); a prompt and a target text per sample
# Output: mean cross-entropy over the target tokens

def training_loss(image, prompt, target_text):
    # Frozen EVA ViT, then the frozen BLIP-2 Q-Former  [§1] [§3]
    # [code: minigpt4/models/minigpt4.py#L118-L137]
    query_outputs = q_former(vit(image))           # (B, 32, d_qformer)

    # Only trainable layer: one linear projection into the LLM space  [§3]
    # [code: minigpt4/models/minigpt4.py#L73-L75]
    image_tokens = projection(query_outputs)       # (B, 32, d_llm)

    # Put the image tokens in front of the text; stage 1 has no prompt  [§3.1]
    # [code: minigpt4/models/minigpt_base.py#L83-L86]
    inputs = concat(prompt_before_image, image_tokens, prompt_after_image)

    # Target is the answer plus the end symbol  [§3.1] [§3.3]
    # [code: minigpt4/models/minigpt_base.py#L252-L262]
    target_tokens = tokenize(target_text + end_symbol)

    # Loss only on the answer tokens; prompt and image positions are ignored
    # [§3.3] [code: minigpt4/models/minigpt_base.py#L290-L297]
    labels = mask_all_except(target_tokens)        # -100 = ignored
    return cross_entropy(llm(inputs), labels)

Hyperparameters

Values not stated in the paper are marked and taken from the official training configs; those are inference from code and are labelled as such.

Results

Author-reported. Setup: a 100-image set (25 per task) judged by human evaluators for the advanced tasks, COCO caption set judged by ChatGPT, and VQA benchmarks AOK-VQA and GQA §4.2 app. A.2.

The paper's qualitative example of incomplete captions before stage 2 is described in §4.3. BLIP-2 fine-tuned on the same stage 2 data still gives short answers and fails the meme and website tasks §4.3. Replacing the stage 2 data with Localized Narratives gives lower-quality, monotonous text §4.3.

Paper vs code

Official repository: Vision-CAIR/MiniGPT-4 at commit d94738a7626ec43eba6c2cddf3cd2043f1a9689a. The paper's abstract links the project page at https://minigpt-4.github.io/, and that page links this repository as "Code". Nothing else was searched.

Discrepancies between the paper and the code:

• Stage 2 steps. Paper: 400 steps with batch 12 §3.3. Config: 5 epochs of 200 iterations, which is 1,000 steps minigpt4_stage2_finetune.yaml.
• Stage 1 batch. Paper: 256 on 4 A100 §3.1. Config: 64 per dataset with world_size: 1 minigpt4_stage1_pretrain.yaml minigpt4_stage1_pretrain.yaml. The config does not show how the 256 is formed.
• Data mixture. Paper: a combined pool with no ratio §3.1. Config: LAION weight 115 and CC+SBU weight 14 minigpt4_stage1_pretrain.yaml minigpt4_stage1_pretrain.yaml.
• Q-Former weights. Paper: the BLIP-2 Q-Former §1. Code: loads the BLIP-2 checkpoint trained with Flan-T5 XXL minigpt4.py. The paper does not name the checkpoint.
• No Q-Former variant. Paper: maps the ViT output directly to the LLM §4.4. Code: drops the CLS token and concatenates each group of four patch tokens into one vector before the projection minigpt4.py.
• Caption end symbol and length. Stage 1 code: newline end symbol and a 32-token cap minigpt_base.py minigpt4.py. Stage 2: ### and 160 tokens minigpt4_stage2_finetune.yaml. The paper gives neither.
• LLM version. Paper: "Vicuna", without size §3. The README lists Vicuna v0 13B and 7B, and a Llama 2 chat variant that the paper does not describe README.md.

Consistent with the code: the stage 2 loss is applied only to the answer tokens, as "we do not calculate the regression loss" for the prompt implies §3.3 minigpt_base.py.

Pitfalls

• Do not skip stage 2. Without it, the authors report repeated or fragmented text in about a third of detailed descriptions and poems §4.3 tab. 3.
• Stage 2 is noisy data. Its targets come from the stage 1 model, then ChatGPT edits, then hard-coded and manual filters §3.2. The ChatGPT version, the number of manual checkers and the agreement between them are not stated.
• Judges are not human for captions. Caption coverage is scored by ChatGPT with one prompt §4.2 app. A.3. The advanced tasks are scored by human evaluators on 100 images, and the number of evaluators is not stated §4.2. Each task has only 25 images, so one image moves a task score by 4 points.
• Hallucination rises with length. Long descriptions (about 175 words) hallucinate more than short ones tab. 5 §4.5. Choose the prompt length on purpose.
• Spatial errors. The model may fail to locate objects in the image, such as windows §4.5. The authors suggest region-grounded data as a fix; this has not been tested in the paper.
• Unfair baselines. BLIP-2 uses Flan-T5 XXL, MiniGPT-4 uses Vicuna §4.1. BLIP-2 is trained on 129M pairs, MiniGPT-4 on about 5M app. A.2. The stage 2 data is also used to fine-tune BLIP-2 as a control §4.3.
• Frozen does not mean free. The ViT and LLM still run a forward pass on every step minigpt_base.py. The projection's output width is set to the LLM's hidden size minigpt4.py, so a different LLM needs a new projection. The paper does not test swapping LLMs.
• Numbers depend on the setup. VQA results lag BLIP-2 app. A.2. The stronger VQA variant changes the recipe: it unfreezes the LLM with LoRA and adds VQA data, so it is not the same method app. A.2.
• Figures. Figure numbers beyond Fig. 1 are counted from the LaTeX source order, not read from a compiled PDF. Cite figures by their captions if a number looks off.

Lineage

• Builds on: BLIP-2 for the vision encoder and Q-Former §1 (its card is not in this wiki); Vicuna, built on LLaMA, as the language model §3.
• Compared against: BLIP-2 §4.2 and BLIP-2 fine-tuned on the same stage 2 data §4.3.
• Related: InstructBLIP also trains a BLIP-2 style model with a Q-Former and a frozen LLM, but it instruction-tunes the Q-Former instead of training one projection.
• Superseded by: not known from content in this wiki.