Improved Baselines with Visual Instruction Tuning

In plain words

A "large multimodal model" is a chatbot that can also look at images. The common recipe takes three ready-made parts: an image encoder that turns a picture into a list of feature vectors, a language model, and a small connector that translates the image features into the language model's input space. LLaVA was one of the simplest designs of this kind, with a single linear layer as the connector. It was good at chatting about images but poor at benchmarks that want a one-word answer.

This paper asks which small changes fix that. There are three. The connector becomes a two-layer neural network instead of one linear layer. The image encoder sees images at a higher resolution. And the training mix gets standard question-answering datasets, each with a short instruction such as "Answer the question using a single word or phrase." That instruction matters: it tells the model when a short answer is wanted, so it doesn't start answering every question with one word.

The resulting model, LLaVA-1.5, uses only public data (about 1.2 million samples in total), trains in about a day on one machine with eight GPUs, and the authors report it beats much larger and more expensively trained systems on most of the twelve benchmarks they test. A variant, LLaVA-1.5-HD, handles larger images by cutting them into tiles the encoder already understands, plus a small overview copy of the whole image.

TL;DR

LLaVA-1.5 is LLaVA with four changes: a 2-layer MLP (GELU) vision-language connector instead of a linear one, CLIP-ViT-L/14 at 336px input, a 665K-sample instruction mix that adds academic VQA/OCR/region datasets with explicit response formatting prompts, plus 40K ShareGPT text conversations, and Vicuna-v1.5 as the LLM. Training is two stages: (1) connector-only alignment on 558K image-caption pairs, (2) full LLM + connector finetune on the 665K mix; the vision encoder stays frozen. 13B takes about 6 h + 20 h on 8×A100. LLaVA-1.5-HD tiles images into 224² grids (up to 672×448) encoded independently, with a downsampled global image concatenated. Read this if you need a reproducible, cheap LMM baseline. The official repo's training scripts reproduce the non-HD model exactly; HD has no v1.5 script.

Problem

Instruction-tuned LMMs split into two camps: LLaVA does well at open-ended visual conversation but poorly on short-answer academic VQA, and it tends to answer yes to yes/no questions because such data is missing §3.1. InstructBLIP does well on VQA but overfits to short answers even when a detailed reply is requested (an example in tab. 1) §3.1. The authors trace this to two causes: (a) ambiguous prompt formats like Q: {Question} A: {Answer} that don't tell the model what length of answer is wanted, and (b) InstructBLIP only tuning the Q-Former, not the LLM, so the visual tokens alone must steer output length §3.2. More broadly, it was unclear which design choices (data amount, resamplers, pretraining scale) actually drive LMM quality §1. The paper runs a controlled study starting from LLaVA §1.

Method

Section numbering follows the compiled paper: §1 Introduction, §2 Related Work, §3 Approach (§3.1 Preliminaries, §3.2 Response Format Prompting, §3.3 Scaling the Data and Model, §3.4 Scaling to Higher Resolutions), §4 Empirical Evaluation (§4.1 Benchmarks, §4.2 Results, §4.3 Emerging Properties, §4.4 Ablation on LLM Choices), §5 Open Problems (§5.1 Data Efficiency, §5.2 Hallucination, §5.3 Compositional Capabilities), §6 Conclusion. Appendix: A Implementation Details (A.1 LLaVA-1.5-HD, A.2 Data, A.3 Hyperparameters), B Qualitative Results, C Limitations. The paper has no numbered equations or algorithms. Tables are numbered 1 to 12 in order; the global-context ablation in §4.2 has no caption and no number.

Architecture fig. 1 §3.3. Three parts: a pretrained vision encoder, a vision-language connector, and an LLM. The visual feature tokens are inserted into the LLM's input sequence where the image placeholder sits.

• Vision encoder: CLIP-ViT-L-336px, chosen as the highest-resolution CLIP available §3.3. The repo uses openai/clip-vit-large-patch14-336 finetune.sh. The code takes hidden states from the second-to-last layer (mm_vision_select_layer -2) and drops the CLS token, keeping only patch tokens finetune.sh clip_encoder.py. The paper states neither. The encoder is frozen in both stages clip_encoder.py.
• Connector: a two-layer MLP replaces LLaVA's single linear projection §3.3. In code (mlp2x_gelu): Linear(d_vision → d_llm) → GELU → Linear(d_llm → d_llm) builder.py.
• LLM: Vicuna v1.5, 7B or 13B app. A.3 tab. 3.

At 336 px with patch size 14 this gives 24×24 = 576 visual tokens per image. That figure is arithmetic from the encoder name, not stated in the paper.

Response format prompting §3.2. VQA questions whose answer should be short get one fixed suffix: "Answer the question using a single word or phrase." Multiple-choice data gets "Answer with the option's letter from the given choices directly." §3.3. The LLM is finetuned on these prompts, so it learns to switch between short and long answers on instruction, and VQA answers need no ChatGPT rewriting §3.2. Per-dataset prompts are listed in the data mixture table below tab. 7.

Training, two stages §3.3 app. A.3 tab. 9.

Stage 1: vision-language alignment pretraining
  data:      558K image-caption pairs, same set as LLaVA          [§3.3] [tab. 3]
             (repo file: blip_laion_cc_sbu_558k.json)            [code: scripts/v1_5/pretrain.sh#L7]
  trainable: connector only (vision encoder and LLM frozen)      [code: llava/train/train.py#L926-L930]
  format:    "plain" template: <image tokens> + caption + "\n";
             loss only on caption tokens                         [code: llava/train/train.py#L588-L607]
  lr 1e-3 (half of LLaVA's, because of the MLP), batch 256, 1 epoch   [app. A.3] [tab. 9]

Stage 2: visual instruction tuning
  data:      665K mixture (tab. 7), concatenated, sampled with equal probability   [app. A.2]
  trainable: connector + full LLM (vision encoder frozen)
  init:      connector weights from stage 1                      [code: scripts/v1_5/finetune.sh#L10]
  format:    Vicuna v1 chat template; loss only on assistant turns   [code: llava/train/train.py#L414-L490]
  images:    padded to square with the CLIP mean colour, then resized   [code: llava/train/train.py#L702-L716]
  batches:   each batch drawn from one modality (image vs text-only)   [app. A.2]
  lr 2e-5, batch 128, 1 epoch                                    [tab. 9]

Instruction data mixture tab. 7 app. A.2.

Data processing rules app. A.2:

1. For every VQA dataset, all QA pairs on the same image are merged into one multi-turn conversation.
2. ShareGPT: invalid conversations are filtered as in Vicuna. Conversations longer than 2048 tokens are truncated, not split into several, which leaves about 40K.
3. A-OKVQA: each QA pair is repeated k times, k = number of choices, to make up for scarce multiple-choice data.
4. OCRVQA: 80K conversations sampled.
5. Visual Genome: 10 annotations sampled for images that have more.
6. RefCOCO: conversations cut into segments of fewer than 10 turns each.
7. Each batch holds a single modality (all multimodal or all text-only). Text conversations are longer, and this grouping makes training 25% faster without changing results.

Ablation path tab. 2. Starting from LLaVA-7B at 224 px, the authors add, in order: VQAv2, the format prompt, the MLP connector, OKVQA/OCR data, region-level VQA, 336 px resolution, GQA, ShareGPT, and finally a 13B LLM. Full numbers are in Results.

LLaVA-1.5-HD §3.4 fig. 2 app. A.1. This variant raises input resolution without interpolating ViT position embeddings and without extra pretraining.

LLaVA-1.5-HD image path   [app. A.1.1]
Encoder: CLIP-ViT-L-14 at 224²
1. Pick a target resolution from grids of up to 6 tiles of 224²:
   1x1, 1x2, 1x3, 1x4, 1x5, 1x6, 2x2, 2x3 and their transposes
   (max 672x448 or 448x672). Criteria: keep as much detail as possible,
   but don't waste pixels (e.g. don't pick 448² for a 224² input).
2. Pad the image to that resolution and split it into 224² tiles.
3. Encode each tile separately with CLIP; stitch the feature maps into one large map.
4. Postprocess:
   a. Drop features that cover only padding.
   b. Append a special row-end token after each row of features,
      so the LLM knows the (variable) image shape.
   c. Flatten to a token sequence.
5. Global context: pad and resize the whole image to 224², encode it, and
   concatenate its features with the high-res features   [§4.2]
Training: no extra pretraining for encoder or connector; visual instruction
tuning runs directly on the high-res inputs   [app. A.1.2]

The paper does not give the exact scoring rule for "detail preservation" vs "resource efficiency", nor the order in which global and tile features are concatenated. See Paper vs code.

Evaluation app. A.3 tab. 8. Greedy decoding. Per-benchmark format prompts:

POPE is reported as F1 on the random/popular/adversarial splits. SEED-Bench video questions use the middle frame. LLaVA-Bench-in-the-Wild and MM-Vet are scored by GPT-4 §4.1.

Hyperparameters

Results

All numbers are author-reported. Unless noted, LLaVA-1.5 uses 558K pretraining and 665K finetuning samples.

Scaling ablation tab. 2 (GQA = short-answer VQA, MME = format-following VQA, MM-Vet = visual conversation; * = GQA training images seen in training):

Rows 0 to 4 use only a subset of InstructBLIP's training data tab. 2. The largest MM-Vet jump comes from moving to 13B §3.3.

Academic-task benchmarks tab. 3 (* = training images/annotations seen; † = includes in-house data):

Instruction-following LMM benchmarks tab. 4:

Global context ablation for HD (7B) §4.2 (uncaptioned table):

Other findings:

• Adding only VQAv2 with the format prompt lifts MME from 809.6 to 1323.8, 111 points above InstructBLIP §3.2 tab. 2.
• VizWiz unanswerable questions: correct "Unanswerable" responses rise from 11.1% to 67.8% with the format prompt §4.3.
• MMBench-CN: 63.6 vs Qwen-VL-Chat's 56.7, with no Chinese multimodal training data; the authors attribute the multilingual ability to ShareGPT §4.3.
• LLM choice §4.4 fig. 3: Vicuna-v1.5 is best overall. LLaMA-2-based models beat LLaMA-1-based ones. LLaMA-2-Chat matches Vicuna-v1.5 on MMBench but is worse on MMBench-CN. Results are shown only as relative-performance radar plots.
• Data efficiency §5.1 fig. 4: random subsampling of the 665K mix (ratios 0.1 to 0.5) keeps more than 98% of full-data performance at 50%. MMBench, ScienceQA and POPE do not drop at 50%, and performance stays steady from 50% down to 30%. Per-benchmark numbers are given only as a plot.
• Hallucination §5.2: hallucination in detailed descriptions drops markedly at 448² input. The authors argue that models learn to hallucinate when training annotations hold detail the input resolution can't resolve. This is a qualitative claim with no table.
• The 7B LLaVA-1.5 beats IDEFICS-80B §4.2.

Paper vs code

Checked at commit c121f04 of haotian-liu/LLaVA (Apache-2.0). This repository also carries later LLaVA-NeXT code.

• Hyperparameters match. The stage-1 and stage-2 values in tab. 9 match scripts/v1_5/pretrain.sh and finetune.sh (lr, warmup, weight decay, epochs, cosine schedule, ZeRO-2/3) pretrain.sh finetune.sh. Global batch sizes 256/128 only hold with 8 GPUs; the GPU count is not in the scripts.

• Unstated in the paper, set in code: penultimate CLIP layer with CLS dropped finetune.sh clip_encoder.py; images padded to square with the CLIP mean colour in stage 2 only train.py finetune.sh; no image start/end or patch tokens finetune.sh; stage-1 "plain" format, image followed directly by caption train.py.

• Single-modality batches. The paper says each batch comes from one modality app. A.2. The code (group_by_modality_length) builds megabatches of world_size × per_device_batch from one modality, length-grouped within each modality, shuffles their order, and then appends one final mixed megabatch of the leftover multimodal and text samples llava_trainer.py.

• LLaVA-1.5-HD is not reproduced by any v1.5 script. The repo's anyres path (used for LLaVA-NeXT) is a related but different mechanism mm_utils.py mm_utils.py llava_arch.py:

  • Tile size equals the encoder's crop size (336 with the v1.5 tower), not the 224 of the paper.
  • The global image is a plain resize to square, with no padding mm_utils.py. The paper says "pad and resize" §4.2.
  • Grid tiles are padded with black mm_utils.py, not the mean colour.
  • Global features come before the tile features llava_arch.py.
  • Resolution choice maximises effective (downscaled) resolution, with ties broken by least wasted area mm_utils.py. This is one concrete rule for the paper's two criteria.
  • Padding removal and the row-end token exist under mm_patch_merge_type containing unpad. The row-end token is a learned vector image_newline, initialised as randn × 1/sqrt(d_llm) llava_arch.py llava_arch.py.

  Which of these choices the paper's HD model used is not stated.

Pitfalls

• The format prompt is the crux. Without it, adding VQAv2 alone gets MME to 1197.0. With it, MME reaches 1323.8 tab. 2. Use the exact strings, and use the same prompts at evaluation tab. 8. A different suffix at eval time is a different experiment.
• Fine-tune the LLM. The paper names Q-Former-only tuning as one cause of InstructBLIP's short-answer overfitting §3.2. Freezing the LLM in stage 2 (e.g. to save memory) brings that failure back.
• Stage-1 lr is 1e-3, not LLaVA's 2e-3. It was halved because of the MLP app. A.3.
• Preprocessing differs between stages in code: no padding in stage 1, mean-colour pad-to-square in stage 2 (see Hyperparameters). The paper mentions neither. Inference has to use stage-2 preprocessing (image_aspect_ratio=pad) to match.
• Penultimate-layer CLIP features, without CLS, come only from the code. Using the last layer or keeping CLS changes the token count (577 vs 576) and the features the connector was trained on.
• Benchmark contamination flags. VQAv2, GQA and TextVQA (for some baselines) are marked * because their training images or annotations were seen in training tab. 3. GQA numbers from row 7 of tab. 2 onward are in-distribution. VizWiz, SciQA, POPE, MME, MMBench, SEED and MM-Vet are not marked.
• GPT-4-judged metrics (LLaVA-Bench-in-the-Wild, MM-Vet) depend on the judge version, which is not stated §4.1.
• ShareGPT truncation at 2048 tokens (not splitting) app. A.2. Each image adds 576 tokens of context in stage 2, and the combined sequence is truncated to the tokenizer max length after the image features are inserted llava_arch.py, so long multi-turn VQA conversations can lose their last turns.
• Cost hidden by "1 day": the stage-2 ~20 h figure is for 13B on 8×A100 with ZeRO-3 §3.3. Training time is about 2× LLaVA's because of the 336² input §3.3. HD increases visual tokens further (up to 6 tiles + global). The paper lists longer training as a limitation app. C and gives no HD training time.
• A-OKVQA augmentation repeats each question k times, but the paper does not say how the copies differ (e.g. whether options are reordered) app. A.2. Check the released mixture file rather than guessing.
• Region data format. For RefCOCO/VG the paper does not specify the bounding-box coordinate format tab. 7. It lives in the released llava_v1_5_mix665k.json (not in the repo) finetune.sh.
• Single image only. The model cannot take multiple images app. C.

Lineage

• Builds on: LLaVA (Liu et al., Visual Instruction Tuning, arXiv:2304.08485), which uses the same overall design with a linear connector, 224² CLIP and LLaVA-Instruct data only.
• Data and format idea contrasted with: InstructBLIP (arXiv:2305.06500), which mixes academic VQA data but uses ambiguous prompts and Q-Former-only tuning.
• Compared against: BLIP-2, InstructBLIP, Shikra, IDEFICS-9B/80B, Qwen-VL / Qwen-VL-Chat, specialist PaLI-X-55B tab. 3 tab. 4.
• Uses: CLIP ViT-L/14-336 (vision-transformer), Vicuna v1.5.