Visual instruction tuning

Overview

A vision-language model has three parts: a vision encoder that turns the image into features, a language model that writes the answer, and a bridge that maps the image features into the language model's input space. A survey of the field uses the same split and adds three training stages: pre-training, instruction tuning and alignment tuning 2306.13549 §2 [arxiv:2306.13549 §3]. Visual instruction tuning is the middle stage, and it trains this stack on image, instruction and answer data, so the model follows a request about an image instead of only describing it.

Where the cards state it, the vision encoder is frozen in LLaVA, MiniGPT-4, InstructBLIP, LLaVA-1.5, Video-ChatGPT, PandaGPT, ViP-LLaVA, Shikra, Otter and GeoChat 2304.08485 §4.2 2304.10592 §1 2305.06500 §2.2 2310.03744 §3.3 2306.05424 §3.2 2305.16355 §3 2312.00784 app. Training Details 2306.15195 §5.1 2305.03726 §3.1 2311.15826 §3.1. DriveMLM does not state whether its visual and LiDAR encoders are frozen [arxiv:2312.09245 §4.2]. The LLaVA-Plus paper does not name its vision encoder in the body 2311.05437 §2. In VideoChat2 the encoder is frozen only in stage 1 2311.17005 §4.2. Emu2 trains its encoder in the first pretraining stage and freezes it in the second 2312.13286 §2.2.2. mPLUG-Owl trains its encoder in stage 1 and freezes it in stage 2 2304.14178 §3.2.

The recipes differ in how much of the rest is trained, in what the bridge is, and in how the instructions are written. LLaVA, InstructBLIP, LLaVA-1.5, MiniGPT-4, PandaGPT, ViP-LLaVA and Otter compute the loss on the response tokens, and so does mPLUG-Owl's second stage 2304.08485 §4.2 2305.06500 §2.2 2310.03744 §3.3 2304.10592 §3.3 2305.16355 §3 2312.00784 §3.1 2304.14178 §3.2 2305.03726 eq. 1. Shikra masks the user tokens in its released code [code: mllm/dataset/process_function/shikra_process_function.py#L115-L131]. LLaVA-Plus gives loss only to assistant spans 2311.05437 §2.1, and DriveMLM applies cross-entropy to its decision and explanation text 2312.09245 §3.3. The newer variants also change what the model is asked to emit: a region answer, a coordinate, a tool call or a driving decision. The main decisions are therefore which parts train, how the bridge reads the input, how the answer's length and format are controlled, and what the answer is for.

Variants

Linear projection, two stages (LLaVA)

The original LLaVA joins a frozen CLIP ViT-L/14 to Vicuna through one trainable matrix W 2304.08485 §4.1 eq. 1. Image features are taken from the layer before the last, and the class token is dropped in the released code 2304.08485 §4.1 tab. 8 clip_encoder.py. The authors chose this linear bridge for speed. They name Flamingo's gated cross-attention and BLIP-2's Q-Former as more elaborate options, and did not test them 2304.08485 §4.1 fig. 1.

Data: 158K samples generated by GPT-4 from COCO captions and object boxes. GPT-4 never sees the image; it reads text that describes it. The split is 58K conversation, 23K detailed description and 77K complex reasoning 2304.08485 §3.

Two stages 2304.08485 §4.2 app. C:

1. Feature alignment: only W trains, on about 595K filtered CC3M image-caption pairs, for one epoch at learning rate 2e-3.
2. End-to-end tuning: W and the LLM train on the 158K data for 3 epochs at learning rate 2e-5. The image comes before or after the first question, at random 2304.08485 eq. 2.

The authors report 85.1 on their COCO benchmark, a relative score from a text-only GPT-4 judge, against 21.5 without instruction tuning 2304.08485 tab. 4. The score is not accuracy. On ScienceQA, the paper reports 90.92 for the model alone 2304.08485 tab. 7.

Card: Visual Instruction Tuning.

Single projection with a curated second stage (MiniGPT-4)

MiniGPT-4 links a frozen BLIP-2 vision encoder and Q-Former to a frozen Vicuna LLM through one linear projection 2304.10592 §3 §1. The Q-Former is not instruction-tuned, which separates it from InstructBLIP 2304.10592 Lineage. The projection is the only part trained in stage 1 2304.10592 §3.1.

Stage 1 trains the projection on about 5M image-caption pairs from Conceptual Captions, SBU and LAION, for 20,000 steps at batch size 256 on 4 A100 GPUs, about ten hours 2304.10592 §3.1 §1. The authors report that this alone gives repeated or fragmented text 2304.10592 §3.1 §4.3.

Stage 2 is the fix. The authors sample 5,000 Conceptual Captions images and ask the stage 1 model for detailed descriptions. ChatGPT repairs the text, hard-coded filters remove apology phrases, and the authors check the rest by hand. About 3,500 pairs remain 2304.10592 §3.2. Stage 2 runs 400 steps at batch size 12, about seven minutes on one A100 2304.10592 §3.3. In the authors' test, the failure rate on detailed descriptions falls from 35% to 2% 2304.10592 tab. 3.

Author-reported: 65 of 100 requests satisfied on four advanced tasks, against 5 for BLIP-2 2304.10592 tab. 1. Caption coverage is 66.2% against 27.5%, judged by ChatGPT 2304.10592 tab. 2.

Card: MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.

Q-Former bridge with a frozen LLM (InstructBLIP)

The model starts from BLIP-2, with a frozen ViT-g/14 image encoder and a frozen LLM (FlanT5 or Vicuna) 2305.06500 §2.3. Only the Q-Former is trained 2305.06500 §2.2. It holds K learnable query embeddings. The instruction tokens join the queries' self-attention, and the queries cross-attend to the image features, so the visual output depends on the task 2305.06500 §2.3. A linear projection then feeds the K outputs to the LLM as soft prompts 2305.06500 §2.3. The paper does not state K. The released config defaults to 32 2305.06500 §2.3.

Data: 26 public datasets in 11 task types, each rewritten as instruction and answer pairs with 10 to 15 hand-written templates per task [arxiv:2305.06500 §2.1]. Thirteen datasets are held in for training and thirteen are held out for zero-shot tests 2305.06500 §2.2. Sampling uses probability proportional to the square root of dataset size, then hand-set adjustments whose values are not given 2305.06500 §2.4.

Setup: 60K steps, batch size 192, 128 or 64 by model, on 16 A100 40GB GPUs in under 1.5 days 2305.06500 §2.6. Finetuning trains 188M parameters, against 1.2B for BLIP-2 finetuning 2305.06500 §3.5. Video input takes four uniformly sampled frames, each encoded separately and concatenated, which gives four times as many visual vectors (derived from 2305.06500 §2.5).

Card: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.

Linear-to-MLP bridge with full LLM tuning (LLaVA-1.5)

The vision encoder is CLIP ViT-L/14 at 336 px, frozen. The bridge is a two-layer MLP instead of a single linear layer. The LLM is Vicuna v1.5, at 7B or 13B 2310.03744 §3.3 2310.03744 app. A.3. At 336 px with 14-pixel patches, this gives 576 visual tokens per image. That count is arithmetic from the encoder's name; the paper does not state it 2310.03744 §3.3.

Two stages 2310.03744 tab. 9:

1. Alignment: only the bridge trains, on 558K image-caption pairs. Learning rate 1e-3, batch 256, one epoch.
2. Instruction tuning: the bridge and the full LLM train on a 665K mixture of public data 2310.03744 app. A.2. Learning rate 2e-5, batch 128, one epoch. Mixture members are sampled with equal probability [arxiv:2310.03744 app. A.2].

The response-format prompts are the core idea. Short-answer VQA questions get the suffix "Answer the question using a single word or phrase.", and multiple-choice questions get a letter-answer suffix 2310.03744 §3.2 2310.03744 tab. 7. The same prompts are used at evaluation 2310.03744 app. A.3 2310.03744 tab. 8. In the ablation, adding the format prompt to VQAv2 data raises MME from 1197.0 to 1323.8 2310.03744 tab. 2.

Compute: 13B on 8 A100 GPUs takes about 6 hours for stage 1 and about 20 hours for stage 2 2310.03744 §3.3. The paper reports that random subsets of 50% of the mixture keep more than 98% of full-data performance 2310.03744 §5.1. The model takes one image per prompt 2310.03744 app. C.

Card: Improved Baselines with Visual Instruction Tuning.

Higher resolution by tiling (LLaVA-1.5-HD)

The image is split into tiles of the encoder's native 224 px, on a grid of up to 6 tiles (up to 672×448, or the transpose). Each tile is encoded separately. A downscaled view of the whole image is encoded too, and its features are added as global context 2310.03744 §3.4 2310.03744 app. A.1 2310.03744 §4.2. There is no extra pretraining; instruction tuning runs directly on the high-resolution inputs 2310.03744 app. A.1.2.

Gains in the reported tables are small. At 13B, TextVQA is 62.5 against 61.3 at 336 px 2310.03744 tab. 3, and MM-Vet is 39.4 against 36.1 2310.03744 tab. 4. Each tile adds visual tokens, and the paper gives no HD training time 2310.03744 app. C. No v1.5 script in the released repository reproduces the HD path (see Contradictions).

Linear projection over pooled video tokens (Video-ChatGPT)

This variant takes video rather than a single image. CLIP ViT-L/14 encodes each frame into a grid of patch features. The features are pooled two ways: one vector per frame, averaged over the grid, and one vector per patch position, averaged over frames. The two sets are stacked into T+N tokens 2306.05424 §3.1 eq. 1. A single linear layer maps them into the embedding space of Vicuna-7B 2306.05424 §3.1 eq. 2. The paper calls this layer an adapter and says more complex designs are possible 2306.05424 §3.1. The language model and the encoder stay frozen; only the layer learns 2306.05424 §3.2.

Data: 100K video-instruction pairs, about 30% from human-assisted annotation on ActivityNet-200 and about 70% from a semi-automatic pipeline that uses captioning and tagging models and GPT-3.5 2306.05424 §4 [arxiv:2306.05424 §5.3]. Setup: 3 epochs, learning rate 2e-5, overall batch size 32, about 3 hours on 8 A100 40GB GPUs 2306.05424 §5.1.

The paper does not state the number of frames or the input resolution. The released code caps frames at 100, samples them at segment midpoints, and pads the temporal tokens to 100 save_spatio_temporal_clip_features.py save_spatio_temporal_clip_features.py. The card's Paper vs code section records these and the other differences.

Results are author-reported and use a GPT-3.5 judge on a 1-5 scale. Video-ChatGPT has the highest correctness score of the four models compared, at 2.40 2306.05424 tab. 1. It also gives 64.9% accuracy on zero-shot MSVD-QA 2306.05424 tab. 2. The paper reports no variance or significance tests.

Card: Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models.

Q-Former video model in three stages (VideoChat2)

VideoChat2 pairs a video encoder (UMT-L in the paper) with a Q-Former and a Vicuna-7B LLM, and trains in three stages 2311.17005 §4.2. The model is described in the MVBench paper, which is the only card for it.

Stage 1 freezes the encoder and trains the Q-Former with contrastive, matching and captioning losses 2311.17005 §4.2. Stage 2 unfreezes the encoder and adds a linear projection that feeds the query tokens into the LLM, trained on more image and video captions 2311.17005 §4.2. Stage 3 adds LoRA to the frozen LLM and tunes the encoder and Q-Former on instruction data. The instruction text, not the question, goes into the Q-Former [arxiv:2311.17005 §4.2] app. A app. B.

Author-reported: 51.1 average on MVBench, against 35.5 for VideoChat 2311.17005 tab. 2. The released code differs from the paper in several training settings; the card's Paper vs code section lists them, including the stage 2 frame count (4 in the paper, 8 in the code) 2311.17005 §4.2 config_7b_stage2.py.

Card: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark.

Learnable abstractor, then LoRA on the LLM (mPLUG-Owl)

mPLUG-Owl gives LLaMA-7B image input through a CLIP ViT-L/14 encoder and a learnable visual abstractor. The abstractor compresses the patch features into a short sequence of visual tokens that the LLM reads with the text 2304.14178 §3.1. Unlike the frozen-encoder recipes above, stage 1 trains the encoder and the abstractor on image-caption pairs from LAION-400M, COYO-700M, Conceptual Captions and MSCOCO, with the LLM frozen. The loss covers text tokens only 2304.14178 §3.2 2304.14178 §4.1.

Stage 2 freezes the whole model except LoRA adapters on the LLM. Its data is 102k Alpaca, 90k Vicuna and 50k Baize text-only examples, plus 150k LLaVA multimodal examples, and the loss is on the response 2304.14178 §3.2 2304.14178 §4.1.

The stage-1 values in the text and in the appendix table disagree, for example learning rate 1e-4 against 2e-4, and 2,000 warm-up steps against 375 2304.14178 §4.1 2304.14178 app. A. The repository releases only the stage-2 entry point, so stage 1 cannot be checked against code train.py. The abstract also says the abstractor is trained with LoRA, while §3.2 and the code freeze it 2304.14178 §1. See Contradictions.

Author-reported: 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4 2304.14178 §4.2. The questions are open-ended and the ratings are manual. The paper does not state how many raters there were or whether they agreed 2304.14178 §4.2.

Card: mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality.

Shared-embedding encoder with LoRA (PandaGPT)

The encoder is ImageBind (huge), frozen. It places image, video, audio, depth, thermal and IMU inputs in one embedding space 2305.16355 §1 2305.16355 §3. A linear projection maps the encoder's single vector into the language model's input space, and Vicuna-13B reads it as one extra token 2305.16355 §3 openllama.py. Vicuna stays frozen except for LoRA on the query, key, value and output projections of its attention 2305.16355 §3 openllama.py. The paper gives the trainable share as about 0.4% of Vicuna's parameters [arxiv:2305.16355 §3].

Data: 160k image-language instruction examples, two epochs, learning rate 5e-4 with linear decay 2305.16355 §3. The loss covers system responses only 2305.16355 §3. Training takes about 7 hours on 8 A100 40GB GPUs 2305.16355 §3.

Other modalities are never trained on. They work because ImageBind places them in the shared space, and several inputs are combined by adding their vectors 2305.16355 §1 2305.16355 §4. The paper reports example outputs only, with no benchmark scores, baselines or ablations 2305.16355 §4. The cross-modal behaviour is therefore a demonstration, not a measurement.

The released PandaGPT data and delta weights are CC BY-NC 4.0, for research use only README.md, "Data Preparation" and "Usage and License Notices". A commercial system must train on data it is allowed to use.

Card: PandaGPT: One Model To Instruction-Follow Them All.

Visual prompts drawn onto the image (ViP-LLaVA)

Instead of a region module or coordinate text, the user's region is drawn onto the image. Eight prompt types are used: rectangle, ellipse, point, triangle, mask, mask contour, arrow and scribble. Each is alpha-blended into the pixels it covers 2312.00784 §3.1 eq. 1. The model is otherwise LLaVA-style: CLIP ViT-L/14 at 336 px, features from several layers concatenated, a LayerNorm and a 2-layer MLP, then Vicuna v1.5 at 7B or 13B 2312.00784 §3.1 2312.00784 §5.1.

Training has three stages 2312.00784 §5.1: connector pre-training on 558k captioned pairs; instruction tuning on LLaVA-1.5 data plus 520k region-prompt examples 2312.00784 §3.2; and a short fine-tune on 26k examples, half of them written by GPT-4V from marked images [arxiv:2312.00784 §3.3]. The region data is drawn from RefCOCOg, PointQA, Visual Genome, Flickr30k Entities, VCR and Visual7W 2312.00784 §3.2. The paper gives about 20 hours for stages 1 and 2 at 7B and 40 hours at 13B, on 8 A100 GPUs 2312.00784 §5.1.

Author-reported: Visual7W 87.91% and PointQA 71.77% at 13B 2312.00784 §5.2, and ViP-Bench 48.3% at 13B with boxes [arxiv:2312.00784 §6]. Against textual coordinates on the same 7B model, the prompts score 70.86 on PointQA against 61.4 [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. On Visual7W the coordinates are slightly higher, at 86.36 against 86.04 2312.00784 app. Comparing Visual Prompts with Coordinates.

The paper's alpha range of [0.5, 1] is not what the released code samples 2312.00784 §5.1 visual_prompt_generator.py. The code also uses a fixed alpha of 128 for Visual7W evaluation [code: scripts/eval/v7w.sh#L8]. Use the code's values if you want to match the released weights, and state the choice.

Card: ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts.

Coordinates written as text (Shikra)

Shikra gives regions and locations to the model as plain text. A frozen CLIP ViT-L/14 gives 256 visual tokens (a 16×16 grid), one trained fully connected layer maps them into Vicuna-7B or Vicuna-13B, and the whole language model is tuned 2306.15195 §5.1. Boxes are written [x_min,y_min,x_max,y_max] and points as [x_center,y_center]. Coordinates are normalised by the image size and kept to three decimal places by default 2306.15195 §5.2. No extra vocabulary, position encoder or detector is added 2306.15195 §5.1 §6.2.

Training data is reorganised public captioning, referring and grounding data, plus 5,922 question-and-answer pairs that GPT-4 wrote from captions and boxes, because GPT-4 was not shown the images 2306.15195 §5.3. Stage 1 trains on the reorganised data. Stage 2 raises the sampling share of LLaVA-Instruct-150K and the generated dialogues to 50% 2306.15195 §5.4. Both stages use AdamW with a cosine schedule, learning rate 2e-5 and global batch 64 on 8 A100 GPUs 2306.15195 §5.4.

Author-reported: 87.01 on RefCOCO val for Shikra-7B 2306.15195 tab. 3, and 85.33 on Visual-7W "which box" PointQA 2306.15195 tab. 4. In a toy comparison trained on referring data only, written numbers beat added coordinate tokens on every split 2306.15195 tab. 2. The paper notes that numbers take more tokens per box, which raises cost in dense scenes 2306.15195 §6.2.

Card: Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic.

Remote-sensing LLaVA-1.5 with task tokens and boxes (GeoChat)

GeoChat keeps the LLaVA-1.5 layout: a CLIP ViT-L/14 encoder, a two-layer MLP adaptor and Vicuna-v1.5 7B 2311.15826 §3.1. It changes three things. The encoder reads 504 x 504 input with interpolated position embeddings, which gives 1296 patch tokens per image 2311.15826 §3.1. Boxes are numbers written in the prompt and in the answer, normalised to [0, 100] with a rotation angle 2311.15826 §3.1. One of three task tokens, [grounding], [identify] or [refer], selects the kind of answer 2311.15826 tab. 1.

The CLIP encoder and the MLP adaptor stay frozen, and LoRA trains the LLM. The adaptor starts from LLaVA-1.5's pre-training 2311.15826 §3.1 2311.15826 §3.2. The paper describes two stages: one epoch on all data, then 1600 more steps on grounding data 2311.15826 §5.1. The released LoRA script runs one epoch and has no second stage [code: scripts/finetune_lora.sh#L24]. The data is about 306k training pairs from aerial detection, scene-classification and VQA sets. A text-only LLM writes the questions and answers from object annotations 2311.15826 tab. 1 2311.15826 §4.

Author-reported zero-shot scene classification: 84.43% on UCMerced and 72.03% on AID, against 68.00 and 51.00 for LLaVA-1.5 2311.15826 tab. 5. The authors' grounding benchmark gives 10.6 overall at IoU 0.5, against 7.6 for MiniGPT-v2 2311.15826 tab. 7. That benchmark is built with the training pipeline, so the grounding scores are in-distribution 2311.15826 §5.4.

Card: GeoChat: Grounded Large Vision-Language Model for Remote Sensing.

Domain VQA data with LoRA on LLaVA-1.5 (Social-LLaVA)

The model is LLaVA-v1.5-7B with LoRA adapters, fine-tuned on a new human-annotated VQA set for robots that move among people [arxiv:2501.09024 §IV-A]. The paper claims no algorithmic novelty; the contribution is the data and a proof-of-concept use of it 2501.09024 §IV-A. The LoRA rank, alpha, target modules, learning rate and optimizer are not stated [arxiv:2501.09024 §IV-A]. The paper gives 15 epochs at batch size 4 on one A100 [arxiv:2501.09024 §IV-A].

Data: about 2K robot-human scenes from the SCAND navigation dataset, with over 40K VQA pairs 2501.09024 §III-B 2501.09024 §I. Each pair covers five aspects: perception, prediction, chain-of-thought, final action and explanation. Context and agent labels use fixed options, and the answers are free text 2501.09024 §III-B. The annotator count is not stated.

Author-reported: fifteen human judges scored 50 items on a 1 to 5 scale. Social-LLaVA is above prompted GPT-4V and Gemini 1.5 Pro on all five task types, for example 4.19 against 2.77 and 3.46 on the final action [arxiv:2501.09024 tab. 1] 2501.09024 §IV-B. The baselines were prompted with a task description and three worked examples, not fine-tuned, so the comparison is not like-for-like 2501.09024 §IV-B. No significance test, variance or inter-rater agreement is reported.

The robot run maps the model's language to three hard-coded motions, straight, left and right 2501.09024 §IV-C. The paper names grounding language into motion as open 2501.09024 §V-E. The paper states no release of the dataset or code, and the card found no repository.

Card: Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces.

Spatial relation questions as fine-tuning data (RoboSpatial)

This variant does not change the bridge. It changes the data: about 3M generated questions on spatial relations, with answers computed from 3D boxes rather than from human labels 2411.16537 §1 2411.16537 §3.2. There are three question types. Context asks for free-space image points in a direction from an anchor object. Compatibility asks whether a virtual box of the target's size fits, and configuration asks whether a directional relation holds between two objects. Both of the last two are True or False [arxiv:2411.16537 §3.1] 2411.16537 §3.2.1 2411.16537 §3.2.2. Each relation is read in the ego, world or object frame 2411.16537 §3.2.1. Distance and counting are left out 2411.16537 §3.1 2411.16537 app. B.2.

Training uses 900,000 of these relations, sampled equally across source datasets, plus 100k grounding question-answer pairs as auxiliary data 2411.16537 app. B.1 2411.16537 app. B.3. The 2D base models are VILA-1.5-8B, LLaVA-NeXT-8B, SpaceLLaVA and RoboPoint, and the 3D models are 3D-LLM and LEO 2411.16537 §4.1.1 2411.16537 app. C.2. The 2D runs use each base repository's "default hyperparameters", and the paper does not state them. The 3D runs use 20 and 10 epochs and keep the best validation checkpoint 2411.16537 app. C.2.

Author-reported on the held-out RoboSpatial-Val split: LLaVA-NeXT rises from 30.3 to 60.5 and VILA from 40.2 to 63.9 2411.16537 tab. 3. The average rises from 38.1 at 100K relations to 60.5 at 900K and 72.4 at the full 3M 2411.16537 tab. 9. For LLaVA-NeXT, the grounding data alone gives 32.4, the spatial data alone 51.8, and both together 60.5 2411.16537 tab. 11. On a robot task suite, LLaVA-NeXT succeeds 52.6% of the time against 23.7% 2411.16537 tab. 6.

Card: RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics.

Driving decisions from a fine-tuned multimodal LLM (DriveMLM)

The model reads four camera views over time, one LiDAR point cloud and a system message with the traffic rules, and answers with one speed decision and one path decision plus an explanation 2312.09245 §3.1 [arxiv:2312.09245 §3.2]. Speed decisions are keep, accelerate, decelerate and stop. Path decisions are follow, left or right change, and left or right borrow [arxiv:2312.09245 §3.2]. A conventional motion planner turns the decisions into a trajectory and controls 2312.09245 §3.1. The paper does not write out the decision definitions, and does not describe the interface to the planner [arxiv:2312.09245 §3.2].

The base is LLaMA-7B with an EVA-CLIP ViT-g/14 image encoder, trained for two epochs at learning rate 5e-5 with batch size 256 2312.09245 §4.2. The training data is 280 hours of CARLA driving across 50k routes and 30 scenarios, with decision labels produced by hand-written rules and explanations refined by annotators 2312.09245 §4.1 2312.09245 tab. 2 [arxiv:2312.09245 §3.4].

Author-reported on CARLA Town05 Long: driving score 76.1 and 0.96 miles per intervention, against 71.4 and 0.76 for Apollo 2312.09245 tab. 4. The open-loop baselines, LLaVA 1.5 and InstructBLIP, were not fine-tuned and received a few examples 2312.09245 §4.2. The datasets are available only on request from the authors, and the official repository has no source code or weights at the card's check. The results therefore cannot be re-run from the paper alone.

Card: DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving.

Tool calls inside a visual chat (LLaVA-Plus)

LLaVA-Plus adds a second kind of dialogue to the LLaVA data. The assistant writes a tool call, receives the tool's output as a new human turn, and then writes the answer 2311.05437 §2.1 eq. 2. Each assistant turn has three fields: thoughts, actions (a JSON list of API name and parameters, empty when no tool is needed) and value 2311.05437 §2.1. Tool outputs are written as text, followed by the original question 2311.05437 §2.2. Most of the data is labelled by GPT-4 2311.05437 §2.2. The image stays in the context for the whole session. Tool-tuned models that the authors compare against receive the image only when a tool is called 2311.05437 §1 2311.05437 §3.

Author-reported: LLaVA-Bench In-the-Wild 69.5 for the All Tools 7B model against 57.1 for LLaVA 2311.05437 tab. 4. At 13B, MM-Vet is 35.0 against 32.5 2311.05437 tab. 6. The 13B comparison mixes sources: the LLaVA row is taken from the MM-Vet paper, while the 7B comparison was rerun by the authors 2311.05437 tab. 6.

Two caveats matter for design. The "All Tools" variant runs the understanding tools on every input and writes their results into the context, so it does not test tool choice 2311.05437 §2.3. The Fly variant, which tests choice, scores lower on In-the-Wild at 59.1, and falls to 48.7 without thoughts 2311.05437 tab. 4. The paper gives no training hyperparameters; the official 7B script uses 3 epochs and learning rate 2e-5 [code: scripts/llava_plus/training_llava_plus_v1.3_7b.sh#L24-L32].

Card: LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents.

Pooled visual embeddings in an interleaved model (Emu2-Chat)

Emu2 is a 37B model that reads images and text as one sequence. Each image is average-pooled to a grid and projected by one linear layer into the language model, with no separate query module 2312.13286 §2.1. Pretraining uses an 8×8 grid, which gives 64 embeddings per image 2312.13286 §2.1. The chat stage uses a 16×16 grid, which gives 256 embeddings per image at 448 px 2312.13286 §2.3.1.

Chat tuning starts from the pretrained model and supervises only the answer tokens. The sequence is a system message, then the user instruction, then the answer 2312.13286 §2.3.1. Academic-task and multimodal chat data use different system messages 2312.13286 app. C.1. Video uses 8, 12 or 16 frames, chosen at random 2312.13286 §2.3.1. The settings are 8,000 steps, global batch 768, peak learning rate 1e-5 with 100 warm-up steps and then cosine decay, and AdamW with β2 0.98 2312.13286 §2.3.1 2312.13286 app. C.2. The freezing rules for this stage are not stated 2312.13286 §2.3.1.

Author-reported: 84.9 on VQAv2, where the mark shows that the model was trained on its training split, and 48.5 on MM-Vet 2312.13286 tab. 2. The same table lists 35.4 for LLaVA-1.5-13B on MM-Vet (see Contradictions). On RefCOCO val, the model scores 90.40 2312.13286 tab. 3.

The pretrained backbone also does few-shot prompting with interleaved examples. It scores 68.8 on VQAv2 at 16 shots, with examples chosen by RICES 2312.13286 tab. 1 2312.13286 app. E.1. At zero shots it is below Flamingo-80B and IDEFICS-80B on VQAv2 2312.13286 tab. 1. The same backbone also predicts image embeddings for generation (Emu2-Gen). That path is not covered here 2312.13286 §2.3.2.

The released code is Apache-2.0 but covers inference only, so the chat recipe is paper-only . The README lists the weights as non-commercial README.md.

Card: Generative Multimodal Models are In-Context Learners.

In-context image and video examples (Otter)

Otter is built on OpenFlamingo-9B, a re-implementation of Flamingo, with an MPT-7B decoder and a CLIP ViT-L/14 encoder 2305.03726 §3.1. A perceiver resampler turns the image or video features into a fixed set of visual tokens, and gated cross-attention layers in the decoder read them. Only these two parts train. The gates keep the language layers unchanged at initialisation 2305.03726 §3.1.

What is new is the data. Each training item is a query plus in-context examples, which are other instruction-response items with their own images. The model reads them before the query 2305.03726 §3.2 [arxiv:2305.03726 §4]. In the LLaVA-derived subsets, the examples are the most similar training items, by CLIP image similarity or by instruction text similarity. The other subsets use sequential images or clips with several instruction-response sets 2305.03726 §4.1. The ablations use two- and four-shot contexts 2305.03726 §6.3 2305.03726 tab. 10.

The data is MIMIC-IT, about 3M instances with 2.2M unique instructions 2305.03726 tab. 1. ChatGPT generates the pairs from captions, boxes or other annotations 2305.03726 §5.1. Training is token-level cross-entropy on the response 2305.03726 eq. 1. The table gives learning rate 1e-5 with cosine decay, six epochs, and a batch of 8 per GPU for images and 4 for video 2305.03726 tab. 11.

Author-reported: Otter-Image-9B scores 69.3 on ScienceQA and 86.5 on POPE 2305.03726 tab. 2. On MME cognition, training with two-shot in-context items raises the two-shot evaluation score to 1430.3, against 1310.7 for the same model trained without them 2305.03726 tab. 10. On zero-shot video QA, Otter-Video scores 77.9 on MSVD-QA against 64.1 for mPLUG-Owl-V, judged by a ChatGPT evaluator 2305.03726 tab. 8 2305.03726 §6.5. The paper gives no variance or repeated runs.

Card: Otter: A Multi-Modal Model with In-Context Instruction Tuning.

When to use which

Implementation sketch

The common core: encode the image, map it to LLM tokens, and compute the loss on the response. Recipes differ in what trains and what the bridge reads.

# Shared training step; the vision encoder is frozen in the recipes that state it
# [arxiv:2304.08485 §4.2] [arxiv:2305.06500 §2.2] [arxiv:2310.03744 §3.3]
# [arxiv:2306.05424 §3.2]
def training_step(image, instruction, response, vision, bridge, llm):
    # Patch or region features from the frozen encoder  [arxiv:2310.03744 §3.3]
    feats = vision(image)                           # (N, d_v)

    # Bridge into the LLM embedding space. The InstructBLIP Q-Former also reads
    # the instruction; the LLaVA matrix, the MLP, MiniGPT-4's projection,
    # PandaGPT's projection, Video-ChatGPT's linear layer, Emu2's pooled
    # projection, mPLUG-Owl's abstractor and Otter's resampler ignore it
    # [arxiv:2305.06500 §2.3] [arxiv:2304.08485 §4.1] [arxiv:2304.10592 §3.1]
    # [arxiv:2306.05424 §3.1] [arxiv:2305.16355 §3] [arxiv:2312.13286 §2.1]
    # [arxiv:2304.14178 §3.1] [arxiv:2305.03726 §3.1]
    visual = bridge(feats, instruction)             # (K, d_llm)

    # Loss on response tokens only  [arxiv:2304.08485 §4.2] [arxiv:2305.06500 §2.2]
    # Train the bridge only (LLaVA stage 1, MiniGPT-4 stage 1, InstructBLIP,
    # Video-ChatGPT), bridge + LLM (LLaVA stage 2, LLaVA-1.5 stage 2) or
    # bridge + LoRA (PandaGPT)  [arxiv:2306.05424 §3.2] [arxiv:2305.16355 §3]
    # [arxiv:2310.03744 §3.3]
    logits = llm(visual, instruction, response)
    return lm_loss(logits, response)

Practical notes from the cards:

• Use the same response-format prompt at training and evaluation 2310.03744 §3.2.
• The LLaVA-1.5 paper does not state its image preprocessing. The released code pads images to square with the CLIP mean colour in stage 2, and not in stage 1 train.py finetune.sh. Inference should use the stage-2 preprocessing.
• The original LLaVA paper gives stage 2 batch size 32. Its full-schedule script uses 16 per device, which is 128 on 8 GPUs with no gradient accumulation 2304.08485 app. C finetune_full_schedule.sh finetune_full_schedule.sh. Choose one and report it.
• MiniGPT-4's paper gives 400 stage 2 steps. Its config runs 5 epochs of 200 iterations, which is 1,000 steps 2304.10592 §3.3 [code: train_configs/minigpt4_stage2_finetune.yaml#L32-L33]. Its stage 1 caption end symbol and length cap are set in code and not in the paper minigpt_base.py.
• Shikra's coordinates are relative to a white square padded around the image, not to the original image transform.py. The paper does not mention the padding, so a resized, non-square input will give different targets unless you pad the same way.
• The InstructBLIP paper gives the sampling formula but not the manual weight values 2305.06500 §2.4. Its repository has no instruction-tuning training config, so the §2.6 settings cannot be checked against code 2305.06500 §2.6.
• Video-ChatGPT's training and inference prompts differ in where the question sits relative to the video tokens. The paper's template puts the instruction first 2306.05424 §3.2. Inference puts the video tokens after the question single_video_inference.py. Pick one placement and use it at both ends.
• LLaVA-Plus's parser needs its literal field tags, "thoughts🤔", "actions🚀" and "value👉", one per line, with the action list as JSON mm_utils.py. The paper names the fields but not the tags 2311.05437 §2.1.
• ViP-LLaVA's prompt stroke widths scale with the image size against a 336-px anchor visual_prompt_generator.py. The paper does not say so, and a resized input will change the mark thickness.
• Emu2's chat wrapper sets max_new_tokens to 10 by default [code: Emu2/emu/chat.py#L47]. Longer answers are cut off unless the caller raises it, and the paper does not state an answer length 2312.13286 app. E.2.
• GeoChat pads non-square images to a square before resizing during training, but its evaluation scripts resize and centre-crop with no padding. The two ends see different image content for non-square inputs. The paper does not mention this train.py [code: geochat/eval/batch_geochat_grounding.py#L86].
• The Emu2 code pads and truncates on the left emu.py. Batched inputs must follow the same convention.
• mPLUG-Owl's training transform includes a random horizontal flip, which can turn a left-right answer into the wrong one. The paper mentions only the resize 2304.14178 §4.1 [code: mPLUG-Owl/pipeline/data_utils/processors/caption_processor.py#L27-L28].
• The mPLUG-Owl stage-2 script sets a global batch of 256, but the value never reaches the trainer. With eight processes and micro-batch 4, the effective batch is 32 2304.14178 §4.1 train_it.sh.
• Otter's table gives six epochs, and the repository's scripts use three. Check which one a run uses before comparing numbers 2305.03726 tab. 11 [code: README.md#L172].

Contradictions

Why InstructBLIP overfits to short answers. LLaVA-1.5 reports that InstructBLIP gives short answers even when a detailed reply is requested. It attributes this to ambiguous prompt formats and to tuning only the Q-Former 2310.03744 §3.1 2310.03744 §3.2. InstructBLIP's own ablations test instruction-aware features and data balancing. They do not test prompt format or LLM tuning 2305.06500 tab. 2. Neither side resolves the question.

ScienceQA numbers for the same model. LLaVA-1.5's table lists InstructBLIP (Vicuna-7B) at 60.5 on SciQA-IMG 2310.03744 tab. 3. That matches InstructBLIP's zero-shot row 2305.06500 tab. 1. InstructBLIP's finetuned row gives 79.5 for the same model 2305.06500 tab. 3. These are different protocols and are not comparable.

Hyperparameters and code. The LLaVA-1.5 hyperparameter table matches the released v1.5 scripts 2310.03744 tab. 9 finetune.sh. The HD variant is not reproduced by any v1.5 script. Its tile size, padding and feature order differ from the paper's description mm_utils.py.

Which parameters train in Video-ChatGPT. The paper says only the linear layer is updated 2306.05424 §3.2. Its released training flags also mark the whole input embedding matrix as trainable video_chatgpt.py train_video_chatgpt.md. The card reads the code and concludes that the gradient reaches only the boundary-token rows. That reading was not run, so check it before relying on it.

Visual7W for the same ViP-LLaVA model. The main table gives 87.91% for ViP-LLaVA-13B 2312.00784 §5.2. The appendix table on GPT-4V data gives 88.28 for the 13B model [arxiv:2312.00784 app. Effect of Optional GPT-4V Region-Level Instruction Data]. The abstract uses the main table. Both numbers are reported and neither is resolved here.

Whether RSVQA-LRBEN is zero-shot for GeoChat. Within the same paper, Table 1 lists 56k LRBEN training pairs, but Table 6 calls the model "not specifically finetuned on target dataset" 2311.15826 tab. 1 [arxiv:2311.15826 tab. 6]. The paper does not explain the difference. Neither side is resolved here, so treat the LRBEN numbers as in-distribution.

Which LiDAR encoder DriveMLM uses. The method section says the LiDAR encoder is randomly initialised 2312.09245 §3.3. The experiments section says a GD-MAE encoder fine-tuned on ONCE is used 2312.09245 §4.2. The paper does not say which one the reported model uses. The ablation finds that point clouds do not help 2312.09245 tab. 5.

Whether LLM judges are acceptable for video. MVBench argues that scoring open answers with an LLM adds bias, and uses multiple-choice items for that reason 2311.17005 §1 2311.17005 §3.2. Its own conversation scores are ChatGPT-judged 2311.17005 tab. 3. Video-ChatGPT's main quality results use a GPT-3.5 judge 2306.05424 §5.2. The cards do not settle which measure is the better one, so compare judged and multiple-choice numbers apart.

LLaVA-1.5-13B on MM-Vet. LLaVA-1.5 reports 36.1 for the 13B model at 336 px 2310.03744 tab. 4. The Emu2 comparison table lists 35.4 for LLaVA-1.5-13B 2312.13286 tab. 2. The cards do not say why the numbers differ. Both are reported and neither is resolved here.

Which LoRA targets mPLUG-Owl trains. The abstract and §1 describe LoRA on the LLM and the abstractor together 2304.14178 §1. §3.2 freezes the abstractor and puts LoRA on the LLM only 2304.14178 §3.2. The released code follows §3.2 and attaches LoRA to the language model's attention projections train.py. Neither side is resolved here.

mPLUG-Owl stage-1 values. The text gives learning rate 1e-4, 2,000 warm-up steps, weight decay 0.1 and a batch of 2.1M tokens 2304.14178 §4.1. The appendix table gives 2e-4, 375 steps, 0.01 and 4,096 image-caption pairs 2304.14178 app. A. The stage-1 code is not in the repository, so the conflict cannot be settled from code.

Whether RoboSpatial hurts general benchmarks. The paper's text says trained models keep or slightly improve performance on every general benchmark 2411.16537 app. D.3. Its table shows MME cognition for LLaVA-NeXT falling from 305.4 to 293.2 2411.16537 tab. 10. Neither side is resolved here, so measure the general score for your own model.

Which parts of Otter train. The paper says only the perceiver resampler and the gated cross-attention train 2305.03726 §3.1. The released code also unfreezes the input embeddings, and the LM head for LLaMA-style decoders modeling_otter.py. The paper also gives 1.4B trainable parameters in the text and 1.3B in Table 2 [arxiv:2305.03726 §3.1] 2305.03726 tab. 2. Neither side is resolved here, so budget memory for the larger set.

Papers

• Visual Instruction Tuning: frozen CLIP, one linear matrix, two stages, 158K GPT-4-generated samples; GPT-4-judged benchmark.
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models: frozen BLIP-2 encoder and Q-Former with one projection; a curated second stage of about 3,500 descriptions.
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning: Q-Former that reads the instruction, frozen LLM, 13 held-in and 13 held-out datasets.
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models: frame-pooled CLIP tokens, one trainable linear layer, 100K video-instruction pairs.
• Improved Baselines with Visual Instruction Tuning: LLaVA-1.5, an MLP bridge with full LLM tuning, response-format prompts, and a high-resolution variant.
• PandaGPT: One Model To Instruction-Follow Them All: ImageBind encoder shared across six input types, a linear projection and LoRA, 160k image-instruction examples; examples only, no benchmarks.
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality: learnable visual abstractor on LLaMA-7B; encoder trained in stage 1, LoRA-only stage 2; author-reported OwlEval ratings.
• Otter: A Multi-Modal Model with In-Context Instruction Tuning: OpenFlamingo-9B with in-context image and video examples from MIMIC-IT; author-reported benchmarks and in-context gains.
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic: coordinates written as text, one linear layer, 5,922 GPT-4 referential dialogues; author-reported referring and PointQA results.
• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts: marks drawn on the image, LLaVA-style model, 520k region-prompt examples; author-reported gains on region benchmarks.
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving: fine-tuned LLM that outputs speed and path decisions for a motion planner; closed-loop results in CARLA, data not public.
• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents: tool-call dialogues in visual instruction tuning, with thoughts, actions and value fields; author-reported benchmark gains.
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark: 20 temporal video tasks as multiple-choice items; also describes the VideoChat2 three-stage training.
• GeoChat: Grounded Large Vision-Language Model for Remote Sensing: LLaVA-1.5 layout with 504 px input, [grounding], [identify] and [refer] task tokens, and LoRA on the LLM; author-reported remote-sensing scene and grounding results.
• Generative Multimodal Models are In-Context Learners: Emu2, a 37B interleaved model with pooled image embeddings; chat tuning at 256 tokens per image; author-reported VQAv2 and MM-Vet; non-commercial weights.
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics: about 3M generated spatial questions from 3D scans used to fine-tune 2D and 3D VLMs; author-reported held-out and robot gains.
• Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces: LLaVA-1.5-7B with LoRA on about 40K human-annotated social-navigation VQA pairs; author-reported human ratings above prompted GPT-4V and Gemini 1.5 Pro.
• A Survey on Multimodal Large Language Models: the encoder, connector and LLM split and the three training stages; no experiments of its own.