Visual Instruction Tuning

In plain words

Chatbots built on language models follow instructions well when they are trained on examples of instructions and good answers. Models that look at pictures have had no such training data, because nobody had written down a large set of "look at this picture, answer this request" examples. This paper fills that gap by asking a text-only language model to write those examples. The model gets a picture described as a list of captions and object boxes, and it writes questions and answers about the scene.

The resulting assistant, LLaVA, has a small trainable bridge that turns image features into tokens the language model can read. Most of the parts are borrowed: a pretrained image encoder and a pretrained language model. First the bridge is trained to line up pictures with their captions, and then the language model and bridge are tuned on the generated conversations.

The authors report strong scores on their own picture-chat benchmark and on a science question benchmark when it is combined with GPT-4. The picture-chat results are judged by GPT-4 itself, so they are relative measures, not absolute accuracy.

TL;DR

LLaVA is a vision-language assistant made by linking a frozen CLIP ViT-L/14 encoder to Vicuna with one trainable linear layer. Training data is 158K instruction samples written by GPT-4 from COCO captions and boxes. Read it for the data recipe, the two-stage training and the GPT-4-judged benchmark. Treat the scores as author-reported and judge-dependent.

Problem

Text-only instruction tuning had improved language models' zero-shot behaviour, but the same idea was little explored for images §1. Image-text pairs exist in large numbers, while instruction-following examples that pair an image with a request and a good answer were scarce, and writing them by hand is slow §3. Vision models before this paper were usually built for one task each, and systems that coordinate several models through an LLM were not trained end to end §2. Open multimodal models such as BLIP-2 and OpenFlamingo were not tuned on vision-language instruction data, and the authors report that they tend to describe an image instead of answering the request §2 §5.1.

Method

The method has three parts: a text-only way to create instruction data from images, an architecture with a small bridge, and a two-stage training schedule.

1. Create instruction data from text. Start from COCO images §3. Each image is turned into text: its captions and its object boxes with labels §3. GPT-4 (preferred to ChatGPT in the authors' early tests) gets this text and a few hand-written seed examples for each type, and writes samples of three types §3:

   • Conversation: multi-turn questions about objects, counts, actions, locations and relative positions, using only questions with definite answers §3.
   • Detailed description: one question is drawn at random from a list of description requests and sent to GPT-4 §3 app. E.
   • Complex reasoning: follow-up questions that need step-by-step logic §3.

   The result is 158K samples: 58K conversation, 23K detailed description and 77K complex reasoning §3. The image itself is never shown to GPT-4; it is only used for reference in the paper's example tab. 1.

2. Architecture. A frozen CLIP ViT-L/14 image encoder gives grid features §4.1. The authors use the output of the layer before the last one §4.1 tab. 8. A single trainable matrix W projects these features into the language model's word-embedding size §4.1 eq. 1. The language model is Vicuna §4.1. The authors chose the linear bridge for speed and mention Flamingo's gated cross-attention and BLIP-2's Q-Former as more elaborate options that they did not test §4.1 fig. 1.

3. Sequence layout and loss. Each sample is a sequence in which the image sits before or after the first question, chosen at random eq. 2. Answers end with a stop marker, written as ### tab. 2. The training loss is the negative log-likelihood of the answer tokens only, given the image and the preceding text eq. 3 §4.2. Image and question tokens are not scored.

4. Stage 1: feature alignment. Filter CC3M down to about 595K image-caption pairs (CC-595K) by noun-phrase frequency app. E §4.2. Turn each pair into a one-turn sample whose question asks for a brief description and whose answer is the original caption §4.2 app. E. Freeze the image encoder and the language model, and train only W §4.2.

5. Stage 2: end-to-end tuning. Keep the image encoder frozen, and train W together with the language model §4.2. Two uses are reported:

   • Chatbot: fine-tune on the 158K samples, sampling the three types with equal probability §4.2. Conversations are multi-turn, and the other two types are single-turn §4.2.
   • ScienceQA: fine-tune on the benchmark's training split, with the question and context as the instruction and the reasoning and answer as the target, as a single turn §4.2 §5.2.

6. ScienceQA ensembling with text-only GPT-4 §5.2. Two combinations are reported: use LLaVA's answer when GPT-4 gives none (complement), or ask GPT-4 for a final answer after seeing both answers (judge) §5.2.

# GPT-assisted data generation  [§3]
# Input:  COCO captions and boxes; hand-written seed examples per type
# Output: instruction samples of three types

def make_samples(captions, boxes, seeds):
    # GPT never sees pixels; the image is described as text  [§3]
    context = captions + boxes_as_text(boxes)
    samples = {}
    for kind in ["conversation", "detail", "reasoning"]:  # three types [§3]
        # Seeds are in-context examples for each type  [§3]
        prompt = seeds[kind] + context
        samples[kind] = gpt4(prompt)  # ChatGPT was tried first  [§3]
    return samples

# Image to visual tokens  [§4.1] [eq. 1] [fig. 1]
# Input:  image (3, H, W)
# Output: visual_tokens (N, d_llm)

def encode_image(image):
    # Frozen CLIP ViT-L/14; keep the output of layer -2  [§4.1] [tab. 8]
    hidden_states = clip_vit(image)  # list of (1 + N, d_clip)
    grid = hidden_states[-2]  # (1 + N, d_clip)
    # Drop the class token  [code: clip_encoder.py#L36-L38]
    patches = grid[1:]  # (N, d_clip)
    # Trainable linear projection into the LLM embedding space  [eq. 1]
    visual_tokens = patches @ W  # (N, d_llm)
    return visual_tokens

# Instruction-tuning loss on answer tokens  [§4.2] [eq. 2] [eq. 3]
# Input:  image, turns = [(question, answer), ...]
# Output: scalar loss

def build_sequence(image, turns):
    # Image and first question in random order  [eq. 2]
    first = turns[0].question
    if random_order():
        head = [image, first]
    else:
        head = [first, image]
    # Each answer ends with the stop marker  [tab. 2]
    sequence = head + [turns[0].answer, STOP]
    for turn in turns[1:]:
        sequence += [turn.question, turn.answer, STOP]
    return sequence

def loss(model, image, sequence, is_target):
    total = 0.0
    for i in range(len(sequence)):
        if is_target[i]:
            # Score only answer tokens, conditioned on the image  [eq. 3]
            total -= log_prob(model, sequence[i], image, sequence[:i])
    # Normalisation over tokens is not stated  [§4.2]
    return total

Key equations:

H_v = W \cdot Z_v, \quad Z_v = g(X_v) eq. 1

p(X_a \mid X_v, X_{instruct}) = \prod_{i=1}^{L} p_\theta(x_i \mid X_v, X_{instruct,<i}, X_{a,<i}) eq. 3

Hyperparameters

Results

Author-reported. Setup: instruction-following is judged by text-only GPT-4 against a reference answer that uses ground-truth captions and boxes §5.1. ScienceQA accuracy is the standard test split §5.2.

The COCO-benchmark figures are relative scores against the text-only GPT-4 reference, not accuracy. Data-ablation rows on that benchmark tab. 4: detail plus complex only 81.9; conversation plus 5% detail plus 10% complex 80.5. ScienceQA design ablations tab. 8: training from scratch without stage 1 gives 85.81; last-layer features give 89.96; predicting the answer first gives 89.77.

Qualitative comparison on one GPT-4-paper example: tab. 3. Challenging In-the-Wild examples and failure notes: tab. 6. Architecture: fig. 1.

Paper vs code

Code checked: haotian-liu/LLaVA at commit c121f0432da27facab705978f83c4ada465e46fd (Apache-2.0). The paper's assets section links an anonymised repository app. D; this card uses the authors' public repository, which the project page links (llava-vl.github.io).

• Fine-tune batch size. Paper: 32 §5 app. C. The fine-tune script uses 16 per device finetune_full_schedule.sh with no gradient accumulation finetune_full_schedule.sh. On 8 GPUs this is a global batch of 128, following the README's formula README.md. Pre-training batch (128) matches the paper.
• Fine-tune data. The 158K dataset and 3 epochs in the paper match finetune_full_schedule.sh and finetune_full_schedule.sh. The default scripts/finetune.sh uses a different file, llava_instruct_80k.json, and 1 epoch finetune.sh finetune.sh.
• Class token. The paper does not say whether the CLS token is removed. The code drops it clip_encoder.py.
• Freezing. Consistent: the image encoder is frozen clip_encoder.py; pre-training trains only the projector train.py.
• Loss masking. Consistent: human-side tokens are masked in the multi-turn template train.py and the caption-only pre-training template masks its prompt train.py.
• Stop marker and prompt. ### matches the paper conversation.py tab. 2. Pre-training sets the prompt version to plain pretrain.sh.
• Optimizer. The paper says Adam. No script sets an optimizer, so the Trainer default applies llava_trainer.py.
• Version. The repository is at a later commit than the paper, and scripts/v1_5/ holds later LLaVA-1.5 scripts that this card does not cover. The scripts cited here are the ones labelled as the original LLaVA.

Pitfalls

• Judge-based scores. The benchmark's reference answers and the judge are both text-only GPT-4 §5.1. The judge sees captions and boxes, not the image. The paper calls the setup consistent but lists its robustness as future work app. A. Do not treat these numbers as accuracy.
• Benchmark is in-distribution and small. The COCO benchmark questions come from the same GPT-4 pipeline as the training data §5.1. The In-the-Wild set has 24 images and 60 questions §5.1. Differences of a few points are noise at this size.
• Table 8 and the text disagree. The text says answer-first reaches 89.77 after 12 epochs and reasoning-first reaches 89.77 after 6, while the main model, which predicts reasons first, scores 90.92 §5.2. Table 8 lists answer-first at 89.77 against a best of 90.92 tab. 8. Use the table value; the text is ambiguous.
• GPT-4 as ensemble member. The 92.53 result uses GPT-4 at inference time §5.2 and needs GPT-4 access. The GPT-4 snapshot is not stated. The paper calls this the first use of GPT-4 for model ensembling §5.2.
• Generation settings. Sampling settings for the GPT-4 data generation (temperature and similar) are not stated in the main text or the parts of the appendix this card read.
• Terms of the generating model. The training data was generated by GPT-4 and ChatGPT §3. The paper does not discuss the provider's terms of use for such output. Check them before generating similar data.
• Released data licence. The project page says the data, code and checkpoints are for research use only, and lists the dataset as CC BY-NC 4.0 [project page llava-vl.github.io]. The code repository is Apache-2.0. The paper is CC BY 4.0. This card uses no dataset content. Do not use the released data for commercial training without a separate licence check. The base-model terms (LLaMA, Vicuna) are not covered here.
• Bag-of-patches failure. The authors report that the model answers "yes" about strawberry yogurt in a fridge that has none, which they read as treating the image as a bag of patches §5.1.
• Sensitive step. Use layer -2 and drop the class token. Taking the last layer costs about one point in the ScienceQA ablation tab. 8.

Lineage

• Builds on: CLIP ViT-L/14 as the image encoder §4.1 and Vicuna as the language model §4.1. Text-only instruction tuning of Alpaca-style models §1.
• Compared against: BLIP-2 and OpenFlamingo on the In-the-Wild benchmark §5.1 tab. 5; LLaMA-Adapter and MM-CoT on ScienceQA §5.2 tab. 7.
• Alternatives it names but does not test: Flamingo's gated cross-attention and BLIP-2's Q-Former as bridges §4.1.
• Follow-up named by the paper: improved baselines with visual instruction tuning, arXiv:2310.03744 §6.