--- kind: concept slug: visual-instruction-tuning title: Visual instruction tuning summary: >- Training a vision-language model on image or video instructions. Variants differ in the bridge and what trains: a linear projection (LLaVA), a Q-Former (InstructBLIP), an MLP with full LLM tuning (LLaVA-1.5), an abstractor (mPLUG-Owl), in-context examples (Otter), LoRA (PandaGPT), marks on images (ViP-LLaVA), coordinates as text (Shikra), tool calls (LLaVA-Plus) and driving decisions (DriveMLM). tags: [multimodal, vision-language, instruction-tuning, in-context-learning, lmm, video, audio, tool-use, autonomous-driving, remote-sensing, robot-navigation, spatial-reasoning] papers: ["2304.08485", "2304.10592", "2304.14178", "2305.03726", "2305.06500", "2305.16355", "2306.05424", "2306.13549", "2306.15195", "2310.03744", "2311.05437", "2311.15826", "2311.17005", "2312.00784", "2312.09245", "2312.13286", "2411.16537", "2501.09024"] related: [multimodal-language-model, vision-transformer, hallucination-mitigation, instruction-tuning, low-rank-adaptation, tool-use, learning-from-demonstration, vision-language-pretraining, benchmark-validity, in-context-learning, text-to-image-generation, visual-grounding] status: draft updated_at: 2026-10-31 --- ## Overview A vision-language model has three parts: a vision encoder that turns the image into features, a language model that writes the answer, and a bridge that maps the image features into the language model's input space. A survey of the field uses the same split and adds three training stages: pre-training, instruction tuning and alignment tuning [arxiv:2306.13549 §2] [arxiv:2306.13549 §3]. Visual instruction tuning is the middle stage, and it trains this stack on image, instruction and answer data, so the model follows a request about an image instead of only describing it. Where the cards state it, the vision encoder is frozen in LLaVA, MiniGPT-4, InstructBLIP, LLaVA-1.5, Video-ChatGPT, PandaGPT, ViP-LLaVA, Shikra, Otter and GeoChat [arxiv:2304.08485 §4.2] [arxiv:2304.10592 §1] [arxiv:2305.06500 §2.2] [arxiv:2310.03744 §3.3] [arxiv:2306.05424 §3.2] [arxiv:2305.16355 §3] [arxiv:2312.00784 app. Training Details] [arxiv:2306.15195 §5.1] [arxiv:2305.03726 §3.1] [arxiv:2311.15826 §3.1]. DriveMLM does not state whether its visual and LiDAR encoders are frozen [arxiv:2312.09245 §4.2]. The LLaVA-Plus paper does not name its vision encoder in the body [arxiv:2311.05437 §2]. In VideoChat2 the encoder is frozen only in stage 1 [arxiv:2311.17005 §4.2]. Emu2 trains its encoder in the first pretraining stage and freezes it in the second [arxiv:2312.13286 §2.2.2]. mPLUG-Owl trains its encoder in stage 1 and freezes it in stage 2 [arxiv:2304.14178 §3.2]. The recipes differ in how much of the rest is trained, in what the bridge is, and in how the instructions are written. LLaVA, InstructBLIP, LLaVA-1.5, MiniGPT-4, PandaGPT, ViP-LLaVA and Otter compute the loss on the response tokens, and so does mPLUG-Owl's second stage [arxiv:2304.08485 §4.2] [arxiv:2305.06500 §2.2] [arxiv:2310.03744 §3.3] [arxiv:2304.10592 §3.3] [arxiv:2305.16355 §3] [arxiv:2312.00784 §3.1] [arxiv:2304.14178 §3.2] [arxiv:2305.03726 eq. 1]. Shikra masks the user tokens in its released code [code: mllm/dataset/process_function/shikra_process_function.py#L115-L131]. LLaVA-Plus gives loss only to assistant spans [arxiv:2311.05437 §2.1], and DriveMLM applies cross-entropy to its decision and explanation text [arxiv:2312.09245 §3.3]. The newer variants also change what the model is asked to emit: a region answer, a coordinate, a tool call or a driving decision. The main decisions are therefore which parts train, how the bridge reads the input, how the answer's length and format are controlled, and what the answer is for. ## Variants ### Linear projection, two stages (LLaVA) The original LLaVA joins a frozen CLIP ViT-L/14 to Vicuna through one trainable matrix W [arxiv:2304.08485 §4.1] [eq. 1]. Image features are taken from the layer before the last, and the class token is dropped in the released code [arxiv:2304.08485 §4.1] [tab. 8] [code: llava/model/multimodal_encoder/clip_encoder.py#L38]. The authors chose this linear bridge for speed. They name Flamingo's gated cross-attention and BLIP-2's Q-Former as more elaborate options, and did not test them [arxiv:2304.08485 §4.1] [fig. 1]. Data: 158K samples generated by GPT-4 from COCO captions and object boxes. GPT-4 never sees the image; it reads text that describes it. The split is 58K conversation, 23K detailed description and 77K complex reasoning [arxiv:2304.08485 §3]. Two stages [arxiv:2304.08485 §4.2] [app. C]: 1. Feature alignment: only W trains, on about 595K filtered CC3M image-caption pairs, for one epoch at learning rate 2e-3. 2. End-to-end tuning: W and the LLM train on the 158K data for 3 epochs at learning rate 2e-5. The image comes before or after the first question, at random [arxiv:2304.08485 eq. 2]. The authors report 85.1 on their COCO benchmark, a relative score from a text-only GPT-4 judge, against 21.5 without instruction tuning [arxiv:2304.08485 tab. 4]. The score is not accuracy. On ScienceQA, the paper reports 90.92 for the model alone [arxiv:2304.08485 tab. 7]. Card: [Visual Instruction Tuning](/papers/2304.08485). ### Single projection with a curated second stage (MiniGPT-4) MiniGPT-4 links a frozen BLIP-2 vision encoder and Q-Former to a frozen Vicuna LLM through one linear projection [arxiv:2304.10592 §3] [§1]. The Q-Former is not instruction-tuned, which separates it from InstructBLIP [arxiv:2304.10592 Lineage]. The projection is the only part trained in stage 1 [arxiv:2304.10592 §3.1]. Stage 1 trains the projection on about 5M image-caption pairs from Conceptual Captions, SBU and LAION, for 20,000 steps at batch size 256 on 4 A100 GPUs, about ten hours [arxiv:2304.10592 §3.1] [§1]. The authors report that this alone gives repeated or fragmented text [arxiv:2304.10592 §3.1] [§4.3]. Stage 2 is the fix. The authors sample 5,000 Conceptual Captions images and ask the stage 1 model for detailed descriptions. ChatGPT repairs the text, hard-coded filters remove apology phrases, and the authors check the rest by hand. About 3,500 pairs remain [arxiv:2304.10592 §3.2]. Stage 2 runs 400 steps at batch size 12, about seven minutes on one A100 [arxiv:2304.10592 §3.3]. In the authors' test, the failure rate on detailed descriptions falls from 35% to 2% [arxiv:2304.10592 tab. 3]. Author-reported: 65 of 100 requests satisfied on four advanced tasks, against 5 for BLIP-2 [arxiv:2304.10592 tab. 1]. Caption coverage is 66.2% against 27.5%, judged by ChatGPT [arxiv:2304.10592 tab. 2]. Card: [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](/papers/2304.10592). ### Q-Former bridge with a frozen LLM (InstructBLIP) The model starts from BLIP-2, with a frozen ViT-g/14 image encoder and a frozen LLM (FlanT5 or Vicuna) [arxiv:2305.06500 §2.3]. Only the Q-Former is trained [arxiv:2305.06500 §2.2]. It holds K learnable query embeddings. The instruction tokens join the queries' self-attention, and the queries cross-attend to the image features, so the visual output depends on the task [arxiv:2305.06500 §2.3]. A linear projection then feeds the K outputs to the LLM as soft prompts [arxiv:2305.06500 §2.3]. The paper does not state K. The released config defaults to 32 [arxiv:2305.06500 §2.3]. Data: 26 public datasets in 11 task types, each rewritten as instruction and answer pairs with 10 to 15 hand-written templates per task [arxiv:2305.06500 §2.1]. Thirteen datasets are held in for training and thirteen are held out for zero-shot tests [arxiv:2305.06500 §2.2]. Sampling uses probability proportional to the square root of dataset size, then hand-set adjustments whose values are not given [arxiv:2305.06500 §2.4]. Setup: 60K steps, batch size 192, 128 or 64 by model, on 16 A100 40GB GPUs in under 1.5 days [arxiv:2305.06500 §2.6]. Finetuning trains 188M parameters, against 1.2B for BLIP-2 finetuning [arxiv:2305.06500 §3.5]. Video input takes four uniformly sampled frames, each encoded separately and concatenated, which gives four times as many visual vectors (derived from [arxiv:2305.06500 §2.5]). Card: [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](/papers/2305.06500). ### Linear-to-MLP bridge with full LLM tuning (LLaVA-1.5) The vision encoder is CLIP ViT-L/14 at 336 px, frozen. The bridge is a two-layer MLP instead of a single linear layer. The LLM is Vicuna v1.5, at 7B or 13B [arxiv:2310.03744 §3.3] [arxiv:2310.03744 app. A.3]. At 336 px with 14-pixel patches, this gives 576 visual tokens per image. That count is arithmetic from the encoder's name; the paper does not state it [arxiv:2310.03744 §3.3]. Two stages [arxiv:2310.03744 tab. 9]: 1. Alignment: only the bridge trains, on 558K image-caption pairs. Learning rate 1e-3, batch 256, one epoch. 2. Instruction tuning: the bridge and the full LLM train on a 665K mixture of public data [arxiv:2310.03744 app. A.2]. Learning rate 2e-5, batch 128, one epoch. Mixture members are sampled with equal probability [arxiv:2310.03744 app. A.2]. The response-format prompts are the core idea. Short-answer VQA questions get the suffix "Answer the question using a single word or phrase.", and multiple-choice questions get a letter-answer suffix [arxiv:2310.03744 §3.2] [arxiv:2310.03744 tab. 7]. The same prompts are used at evaluation [arxiv:2310.03744 app. A.3] [arxiv:2310.03744 tab. 8]. In the ablation, adding the format prompt to VQAv2 data raises MME from 1197.0 to 1323.8 [arxiv:2310.03744 tab. 2]. Compute: 13B on 8 A100 GPUs takes about 6 hours for stage 1 and about 20 hours for stage 2 [arxiv:2310.03744 §3.3]. The paper reports that random subsets of 50% of the mixture keep more than 98% of full-data performance [arxiv:2310.03744 §5.1]. The model takes one image per prompt [arxiv:2310.03744 app. C]. Card: [Improved Baselines with Visual Instruction Tuning](/papers/2310.03744). ### Higher resolution by tiling (LLaVA-1.5-HD) The image is split into tiles of the encoder's native 224 px, on a grid of up to 6 tiles (up to 672×448, or the transpose). Each tile is encoded separately. A downscaled view of the whole image is encoded too, and its features are added as global context [arxiv:2310.03744 §3.4] [arxiv:2310.03744 app. A.1] [arxiv:2310.03744 §4.2]. There is no extra pretraining; instruction tuning runs directly on the high-resolution inputs [arxiv:2310.03744 app. A.1.2]. Gains in the reported tables are small. At 13B, TextVQA is 62.5 against 61.3 at 336 px [arxiv:2310.03744 tab. 3], and MM-Vet is 39.4 against 36.1 [arxiv:2310.03744 tab. 4]. Each tile adds visual tokens, and the paper gives no HD training time [arxiv:2310.03744 app. C]. No v1.5 script in the released repository reproduces the HD path (see Contradictions). ### Linear projection over pooled video tokens (Video-ChatGPT) This variant takes video rather than a single image. CLIP ViT-L/14 encodes each frame into a grid of patch features. The features are pooled two ways: one vector per frame, averaged over the grid, and one vector per patch position, averaged over frames. The two sets are stacked into T+N tokens [arxiv:2306.05424 §3.1] [eq. 1]. A single linear layer maps them into the embedding space of Vicuna-7B [arxiv:2306.05424 §3.1] [eq. 2]. The paper calls this layer an adapter and says more complex designs are possible [arxiv:2306.05424 §3.1]. The language model and the encoder stay frozen; only the layer learns [arxiv:2306.05424 §3.2]. Data: 100K video-instruction pairs, about 30% from human-assisted annotation on ActivityNet-200 and about 70% from a semi-automatic pipeline that uses captioning and tagging models and GPT-3.5 [arxiv:2306.05424 §4] [arxiv:2306.05424 §5.3]. Setup: 3 epochs, learning rate 2e-5, overall batch size 32, about 3 hours on 8 A100 40GB GPUs [arxiv:2306.05424 §5.1]. The paper does not state the number of frames or the input resolution. The released code caps frames at 100, samples them at segment midpoints, and pads the temporal tokens to 100 [code: scripts/save_spatio_temporal_clip_features.py#L13-L17] [code: scripts/save_spatio_temporal_clip_features.py#L49-L53]. The card's Paper vs code section records these and the other differences. Results are author-reported and use a GPT-3.5 judge on a 1-5 scale. Video-ChatGPT has the highest correctness score of the four models compared, at 2.40 [arxiv:2306.05424 tab. 1]. It also gives 64.9% accuracy on zero-shot MSVD-QA [arxiv:2306.05424 tab. 2]. The paper reports no variance or significance tests. Card: [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](/papers/2306.05424). ### Q-Former video model in three stages (VideoChat2) VideoChat2 pairs a video encoder (UMT-L in the paper) with a Q-Former and a Vicuna-7B LLM, and trains in three stages [arxiv:2311.17005 §4.2]. The model is described in the MVBench paper, which is the only card for it. Stage 1 freezes the encoder and trains the Q-Former with contrastive, matching and captioning losses [arxiv:2311.17005 §4.2]. Stage 2 unfreezes the encoder and adds a linear projection that feeds the query tokens into the LLM, trained on more image and video captions [arxiv:2311.17005 §4.2]. Stage 3 adds LoRA to the frozen LLM and tunes the encoder and Q-Former on instruction data. The instruction text, not the question, goes into the Q-Former [arxiv:2311.17005 §4.2] [app. A] [app. B]. Author-reported: 51.1 average on MVBench, against 35.5 for VideoChat [arxiv:2311.17005 tab. 2]. The released code differs from the paper in several training settings; the card's Paper vs code section lists them, including the stage 2 frame count (4 in the paper, 8 in the code) [arxiv:2311.17005 §4.2] [code: video_chat2/scripts/videochat_vicuna/config_7b_stage2.py#L13]. Card: [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](/papers/2311.17005). ### Learnable abstractor, then LoRA on the LLM (mPLUG-Owl) mPLUG-Owl gives LLaMA-7B image input through a CLIP ViT-L/14 encoder and a learnable visual abstractor. The abstractor compresses the patch features into a short sequence of visual tokens that the LLM reads with the text [arxiv:2304.14178 §3.1]. Unlike the frozen-encoder recipes above, stage 1 trains the encoder and the abstractor on image-caption pairs from LAION-400M, COYO-700M, Conceptual Captions and MSCOCO, with the LLM frozen. The loss covers text tokens only [arxiv:2304.14178 §3.2] [arxiv:2304.14178 §4.1]. Stage 2 freezes the whole model except LoRA adapters on the LLM. Its data is 102k Alpaca, 90k Vicuna and 50k Baize text-only examples, plus 150k LLaVA multimodal examples, and the loss is on the response [arxiv:2304.14178 §3.2] [arxiv:2304.14178 §4.1]. The stage-1 values in the text and in the appendix table disagree, for example learning rate 1e-4 against 2e-4, and 2,000 warm-up steps against 375 [arxiv:2304.14178 §4.1] [arxiv:2304.14178 app. A]. The repository releases only the stage-2 entry point, so stage 1 cannot be checked against code [code: mPLUG-Owl/pipeline/train.py]. The abstract also says the abstractor is trained with LoRA, while §3.2 and the code freeze it [arxiv:2304.14178 §1]. See Contradictions. Author-reported: 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4 [arxiv:2304.14178 §4.2]. The questions are open-ended and the ratings are manual. The paper does not state how many raters there were or whether they agreed [arxiv:2304.14178 §4.2]. Card: [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](/papers/2304.14178). ### Shared-embedding encoder with LoRA (PandaGPT) The encoder is ImageBind (huge), frozen. It places image, video, audio, depth, thermal and IMU inputs in one embedding space [arxiv:2305.16355 §1] [arxiv:2305.16355 §3]. A linear projection maps the encoder's single vector into the language model's input space, and Vicuna-13B reads it as one extra token [arxiv:2305.16355 §3] [code: code/model/openllama.py#L113-L115]. Vicuna stays frozen except for LoRA on the query, key, value and output projections of its attention [arxiv:2305.16355 §3] [code: code/model/openllama.py#L101]. The paper gives the trainable share as about 0.4% of Vicuna's parameters [arxiv:2305.16355 §3]. Data: 160k image-language instruction examples, two epochs, learning rate 5e-4 with linear decay [arxiv:2305.16355 §3]. The loss covers system responses only [arxiv:2305.16355 §3]. Training takes about 7 hours on 8 A100 40GB GPUs [arxiv:2305.16355 §3]. Other modalities are never trained on. They work because ImageBind places them in the shared space, and several inputs are combined by adding their vectors [arxiv:2305.16355 §1] [arxiv:2305.16355 §4]. The paper reports example outputs only, with no benchmark scores, baselines or ablations [arxiv:2305.16355 §4]. The cross-modal behaviour is therefore a demonstration, not a measurement. The released PandaGPT data and delta weights are CC BY-NC 4.0, for research use only [code: README.md, "Data Preparation" and "Usage and License Notices"]. A commercial system must train on data it is allowed to use. Card: [PandaGPT: One Model To Instruction-Follow Them All](/papers/2305.16355). ### Visual prompts drawn onto the image (ViP-LLaVA) Instead of a region module or coordinate text, the user's region is drawn onto the image. Eight prompt types are used: rectangle, ellipse, point, triangle, mask, mask contour, arrow and scribble. Each is alpha-blended into the pixels it covers [arxiv:2312.00784 §3.1] [eq. 1]. The model is otherwise LLaVA-style: CLIP ViT-L/14 at 336 px, features from several layers concatenated, a LayerNorm and a 2-layer MLP, then Vicuna v1.5 at 7B or 13B [arxiv:2312.00784 §3.1] [arxiv:2312.00784 §5.1]. Training has three stages [arxiv:2312.00784 §5.1]: connector pre-training on 558k captioned pairs; instruction tuning on LLaVA-1.5 data plus 520k region-prompt examples [arxiv:2312.00784 §3.2]; and a short fine-tune on 26k examples, half of them written by GPT-4V from marked images [arxiv:2312.00784 §3.3]. The region data is drawn from RefCOCOg, PointQA, Visual Genome, Flickr30k Entities, VCR and Visual7W [arxiv:2312.00784 §3.2]. The paper gives about 20 hours for stages 1 and 2 at 7B and 40 hours at 13B, on 8 A100 GPUs [arxiv:2312.00784 §5.1]. Author-reported: Visual7W 87.91% and PointQA 71.77% at 13B [arxiv:2312.00784 §5.2], and ViP-Bench 48.3% at 13B with boxes [arxiv:2312.00784 §6]. Against textual coordinates on the same 7B model, the prompts score 70.86 on PointQA against 61.4 [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. On Visual7W the coordinates are slightly higher, at 86.36 against 86.04 [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. The paper's alpha range of [0.5, 1] is not what the released code samples [arxiv:2312.00784 §5.1] [code: llava/visual_prompt_generator.py#L277-L278]. The code also uses a fixed alpha of 128 for Visual7W evaluation [code: scripts/eval/v7w.sh#L8]. Use the code's values if you want to match the released weights, and state the choice. Card: [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](/papers/2312.00784). ### Coordinates written as text (Shikra) Shikra gives regions and locations to the model as plain text. A frozen CLIP ViT-L/14 gives 256 visual tokens (a 16×16 grid), one trained fully connected layer maps them into Vicuna-7B or Vicuna-13B, and the whole language model is tuned [arxiv:2306.15195 §5.1]. Boxes are written `[x_min,y_min,x_max,y_max]` and points as `[x_center,y_center]`. Coordinates are normalised by the image size and kept to three decimal places by default [arxiv:2306.15195 §5.2]. No extra vocabulary, position encoder or detector is added [arxiv:2306.15195 §5.1] [§6.2]. Training data is reorganised public captioning, referring and grounding data, plus 5,922 question-and-answer pairs that GPT-4 wrote from captions and boxes, because GPT-4 was not shown the images [arxiv:2306.15195 §5.3]. Stage 1 trains on the reorganised data. Stage 2 raises the sampling share of LLaVA-Instruct-150K and the generated dialogues to 50% [arxiv:2306.15195 §5.4]. Both stages use AdamW with a cosine schedule, learning rate 2e-5 and global batch 64 on 8 A100 GPUs [arxiv:2306.15195 §5.4]. Author-reported: 87.01 on RefCOCO val for Shikra-7B [arxiv:2306.15195 tab. 3], and 85.33 on Visual-7W "which box" PointQA [arxiv:2306.15195 tab. 4]. In a toy comparison trained on referring data only, written numbers beat added coordinate tokens on every split [arxiv:2306.15195 tab. 2]. The paper notes that numbers take more tokens per box, which raises cost in dense scenes [arxiv:2306.15195 §6.2]. Card: [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](/papers/2306.15195). ### Remote-sensing LLaVA-1.5 with task tokens and boxes (GeoChat) GeoChat keeps the LLaVA-1.5 layout: a CLIP ViT-L/14 encoder, a two-layer MLP adaptor and Vicuna-v1.5 7B [arxiv:2311.15826 §3.1]. It changes three things. The encoder reads 504 x 504 input with interpolated position embeddings, which gives 1296 patch tokens per image [arxiv:2311.15826 §3.1]. Boxes are numbers written in the prompt and in the answer, normalised to [0, 100] with a rotation angle [arxiv:2311.15826 §3.1]. One of three task tokens, `[grounding]`, `[identify]` or `[refer]`, selects the kind of answer [arxiv:2311.15826 tab. 1]. The CLIP encoder and the MLP adaptor stay frozen, and LoRA trains the LLM. The adaptor starts from LLaVA-1.5's pre-training [arxiv:2311.15826 §3.1] [arxiv:2311.15826 §3.2]. The paper describes two stages: one epoch on all data, then 1600 more steps on grounding data [arxiv:2311.15826 §5.1]. The released LoRA script runs one epoch and has no second stage [code: scripts/finetune_lora.sh#L24]. The data is about 306k training pairs from aerial detection, scene-classification and VQA sets. A text-only LLM writes the questions and answers from object annotations [arxiv:2311.15826 tab. 1] [arxiv:2311.15826 §4]. Author-reported zero-shot scene classification: 84.43% on UCMerced and 72.03% on AID, against 68.00 and 51.00 for LLaVA-1.5 [arxiv:2311.15826 tab. 5]. The authors' grounding benchmark gives 10.6 overall at IoU 0.5, against 7.6 for MiniGPT-v2 [arxiv:2311.15826 tab. 7]. That benchmark is built with the training pipeline, so the grounding scores are in-distribution [arxiv:2311.15826 §5.4]. Card: [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](/papers/2311.15826). ### Domain VQA data with LoRA on LLaVA-1.5 (Social-LLaVA) The model is LLaVA-v1.5-7B with LoRA adapters, fine-tuned on a new human-annotated VQA set for robots that move among people [arxiv:2501.09024 §IV-A]. The paper claims no algorithmic novelty; the contribution is the data and a proof-of-concept use of it [arxiv:2501.09024 §IV-A]. The LoRA rank, alpha, target modules, learning rate and optimizer are not stated [arxiv:2501.09024 §IV-A]. The paper gives 15 epochs at batch size 4 on one A100 [arxiv:2501.09024 §IV-A]. Data: about 2K robot-human scenes from the SCAND navigation dataset, with over 40K VQA pairs [arxiv:2501.09024 §III-B] [arxiv:2501.09024 §I]. Each pair covers five aspects: perception, prediction, chain-of-thought, final action and explanation. Context and agent labels use fixed options, and the answers are free text [arxiv:2501.09024 §III-B]. The annotator count is not stated. Author-reported: fifteen human judges scored 50 items on a 1 to 5 scale. Social-LLaVA is above prompted GPT-4V and Gemini 1.5 Pro on all five task types, for example 4.19 against 2.77 and 3.46 on the final action [arxiv:2501.09024 tab. 1] [arxiv:2501.09024 §IV-B]. The baselines were prompted with a task description and three worked examples, not fine-tuned, so the comparison is not like-for-like [arxiv:2501.09024 §IV-B]. No significance test, variance or inter-rater agreement is reported. The robot run maps the model's language to three hard-coded motions, straight, left and right [arxiv:2501.09024 §IV-C]. The paper names grounding language into motion as open [arxiv:2501.09024 §V-E]. The paper states no release of the dataset or code, and the card found no repository. Card: [Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces](/papers/2501.09024). ### Spatial relation questions as fine-tuning data (RoboSpatial) This variant does not change the bridge. It changes the data: about 3M generated questions on spatial relations, with answers computed from 3D boxes rather than from human labels [arxiv:2411.16537 §1] [arxiv:2411.16537 §3.2]. There are three question types. Context asks for free-space image points in a direction from an anchor object. Compatibility asks whether a virtual box of the target's size fits, and configuration asks whether a directional relation holds between two objects. Both of the last two are True or False [arxiv:2411.16537 §3.1] [arxiv:2411.16537 §3.2.1] [arxiv:2411.16537 §3.2.2]. Each relation is read in the ego, world or object frame [arxiv:2411.16537 §3.2.1]. Distance and counting are left out [arxiv:2411.16537 §3.1] [arxiv:2411.16537 app. B.2]. Training uses 900,000 of these relations, sampled equally across source datasets, plus 100k grounding question-answer pairs as auxiliary data [arxiv:2411.16537 app. B.1] [arxiv:2411.16537 app. B.3]. The 2D base models are VILA-1.5-8B, LLaVA-NeXT-8B, SpaceLLaVA and RoboPoint, and the 3D models are 3D-LLM and LEO [arxiv:2411.16537 §4.1.1] [arxiv:2411.16537 app. C.2]. The 2D runs use each base repository's "default hyperparameters", and the paper does not state them. The 3D runs use 20 and 10 epochs and keep the best validation checkpoint [arxiv:2411.16537 app. C.2]. Author-reported on the held-out RoboSpatial-Val split: LLaVA-NeXT rises from 30.3 to 60.5 and VILA from 40.2 to 63.9 [arxiv:2411.16537 tab. 3]. The average rises from 38.1 at 100K relations to 60.5 at 900K and 72.4 at the full 3M [arxiv:2411.16537 tab. 9]. For LLaVA-NeXT, the grounding data alone gives 32.4, the spatial data alone 51.8, and both together 60.5 [arxiv:2411.16537 tab. 11]. On a robot task suite, LLaVA-NeXT succeeds 52.6% of the time against 23.7% [arxiv:2411.16537 tab. 6]. Card: [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](/papers/2411.16537). ### Driving decisions from a fine-tuned multimodal LLM (DriveMLM) The model reads four camera views over time, one LiDAR point cloud and a system message with the traffic rules, and answers with one speed decision and one path decision plus an explanation [arxiv:2312.09245 §3.1] [arxiv:2312.09245 §3.2]. Speed decisions are keep, accelerate, decelerate and stop. Path decisions are follow, left or right change, and left or right borrow [arxiv:2312.09245 §3.2]. A conventional motion planner turns the decisions into a trajectory and controls [arxiv:2312.09245 §3.1]. The paper does not write out the decision definitions, and does not describe the interface to the planner [arxiv:2312.09245 §3.2]. The base is LLaMA-7B with an EVA-CLIP ViT-g/14 image encoder, trained for two epochs at learning rate 5e-5 with batch size 256 [arxiv:2312.09245 §4.2]. The training data is 280 hours of CARLA driving across 50k routes and 30 scenarios, with decision labels produced by hand-written rules and explanations refined by annotators [arxiv:2312.09245 §4.1] [arxiv:2312.09245 tab. 2] [arxiv:2312.09245 §3.4]. Author-reported on CARLA Town05 Long: driving score 76.1 and 0.96 miles per intervention, against 71.4 and 0.76 for Apollo [arxiv:2312.09245 tab. 4]. The open-loop baselines, LLaVA 1.5 and InstructBLIP, were not fine-tuned and received a few examples [arxiv:2312.09245 §4.2]. The datasets are available only on request from the authors, and the official repository has no source code or weights at the card's check. The results therefore cannot be re-run from the paper alone. Card: [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](/papers/2312.09245). ### Tool calls inside a visual chat (LLaVA-Plus) LLaVA-Plus adds a second kind of dialogue to the LLaVA data. The assistant writes a tool call, receives the tool's output as a new human turn, and then writes the answer [arxiv:2311.05437 §2.1] [eq. 2]. Each assistant turn has three fields: `thoughts`, `actions` (a JSON list of API name and parameters, empty when no tool is needed) and `value` [arxiv:2311.05437 §2.1]. Tool outputs are written as text, followed by the original question [arxiv:2311.05437 §2.2]. Most of the data is labelled by GPT-4 [arxiv:2311.05437 §2.2]. The image stays in the context for the whole session. Tool-tuned models that the authors compare against receive the image only when a tool is called [arxiv:2311.05437 §1] [arxiv:2311.05437 §3]. Author-reported: LLaVA-Bench In-the-Wild 69.5 for the All Tools 7B model against 57.1 for LLaVA [arxiv:2311.05437 tab. 4]. At 13B, MM-Vet is 35.0 against 32.5 [arxiv:2311.05437 tab. 6]. The 13B comparison mixes sources: the LLaVA row is taken from the MM-Vet paper, while the 7B comparison was rerun by the authors [arxiv:2311.05437 tab. 6]. Two caveats matter for design. The "All Tools" variant runs the understanding tools on every input and writes their results into the context, so it does not test tool choice [arxiv:2311.05437 §2.3]. The Fly variant, which tests choice, scores lower on In-the-Wild at 59.1, and falls to 48.7 without `thoughts` [arxiv:2311.05437 tab. 4]. The paper gives no training hyperparameters; the official 7B script uses 3 epochs and learning rate 2e-5 [code: scripts/llava_plus/training_llava_plus_v1.3_7b.sh#L24-L32]. Card: [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](/papers/2311.05437). ### Pooled visual embeddings in an interleaved model (Emu2-Chat) Emu2 is a 37B model that reads images and text as one sequence. Each image is average-pooled to a grid and projected by one linear layer into the language model, with no separate query module [arxiv:2312.13286 §2.1]. Pretraining uses an 8×8 grid, which gives 64 embeddings per image [arxiv:2312.13286 §2.1]. The chat stage uses a 16×16 grid, which gives 256 embeddings per image at 448 px [arxiv:2312.13286 §2.3.1]. Chat tuning starts from the pretrained model and supervises only the answer tokens. The sequence is a system message, then the user instruction, then the answer [arxiv:2312.13286 §2.3.1]. Academic-task and multimodal chat data use different system messages [arxiv:2312.13286 app. C.1]. Video uses 8, 12 or 16 frames, chosen at random [arxiv:2312.13286 §2.3.1]. The settings are 8,000 steps, global batch 768, peak learning rate 1e-5 with 100 warm-up steps and then cosine decay, and AdamW with β2 0.98 [arxiv:2312.13286 §2.3.1] [arxiv:2312.13286 app. C.2]. The freezing rules for this stage are not stated [arxiv:2312.13286 §2.3.1]. Author-reported: 84.9 on VQAv2, where the mark shows that the model was trained on its training split, and 48.5 on MM-Vet [arxiv:2312.13286 tab. 2]. The same table lists 35.4 for LLaVA-1.5-13B on MM-Vet (see Contradictions). On RefCOCO val, the model scores 90.40 [arxiv:2312.13286 tab. 3]. The pretrained backbone also does few-shot prompting with interleaved examples. It scores 68.8 on VQAv2 at 16 shots, with examples chosen by RICES [arxiv:2312.13286 tab. 1] [arxiv:2312.13286 app. E.1]. At zero shots it is below Flamingo-80B and IDEFICS-80B on VQAv2 [arxiv:2312.13286 tab. 1]. The same backbone also predicts image embeddings for generation (Emu2-Gen). That path is not covered here [arxiv:2312.13286 §2.3.2]. The released code is Apache-2.0 but covers inference only, so the chat recipe is paper-only [code: Emu2/]. The README lists the weights as non-commercial [code: Emu2/README.md#L80-L82]. Card: [Generative Multimodal Models are In-Context Learners](/papers/2312.13286). ### In-context image and video examples (Otter) Otter is built on OpenFlamingo-9B, a re-implementation of Flamingo, with an MPT-7B decoder and a CLIP ViT-L/14 encoder [arxiv:2305.03726 §3.1]. A perceiver resampler turns the image or video features into a fixed set of visual tokens, and gated cross-attention layers in the decoder read them. Only these two parts train. The gates keep the language layers unchanged at initialisation [arxiv:2305.03726 §3.1]. What is new is the data. Each training item is a query plus in-context examples, which are other instruction-response items with their own images. The model reads them before the query [arxiv:2305.03726 §3.2] [arxiv:2305.03726 §4]. In the LLaVA-derived subsets, the examples are the most similar training items, by CLIP image similarity or by instruction text similarity. The other subsets use sequential images or clips with several instruction-response sets [arxiv:2305.03726 §4.1]. The ablations use two- and four-shot contexts [arxiv:2305.03726 §6.3] [arxiv:2305.03726 tab. 10]. The data is MIMIC-IT, about 3M instances with 2.2M unique instructions [arxiv:2305.03726 tab. 1]. ChatGPT generates the pairs from captions, boxes or other annotations [arxiv:2305.03726 §5.1]. Training is token-level cross-entropy on the response [arxiv:2305.03726 eq. 1]. The table gives learning rate 1e-5 with cosine decay, six epochs, and a batch of 8 per GPU for images and 4 for video [arxiv:2305.03726 tab. 11]. Author-reported: Otter-Image-9B scores 69.3 on ScienceQA and 86.5 on POPE [arxiv:2305.03726 tab. 2]. On MME cognition, training with two-shot in-context items raises the two-shot evaluation score to 1430.3, against 1310.7 for the same model trained without them [arxiv:2305.03726 tab. 10]. On zero-shot video QA, Otter-Video scores 77.9 on MSVD-QA against 64.1 for mPLUG-Owl-V, judged by a ChatGPT evaluator [arxiv:2305.03726 tab. 8] [arxiv:2305.03726 §6.5]. The paper gives no variance or repeated runs. Card: [Otter: A Multi-Modal Model with In-Context Instruction Tuning](/papers/2305.03726). ## When to use which | Situation | Use | Why | |---|---|---| | You want the reference recipe that most later variants build on | LLaVA recipe | One trainable matrix and two stages on 158K generated samples [arxiv:2304.08485 §4.2] [arxiv:2304.08485 §3]. Its benchmark is judged by GPT-4 and is not accuracy [arxiv:2304.08485 tab. 4] | | You need a general chat model from frozen parts, on few GPUs | MiniGPT-4 recipe | Only a projection trains: about ten hours on 4 A100 GPUs for stage 1, then about seven minutes on one A100 for stage 2 [arxiv:2304.10592 §3.1] [arxiv:2304.10592 §3.3]. Skip stage 2 and about a third of detailed descriptions repeat or break [arxiv:2304.10592 tab. 3] | | The LLM can't be updated (compute or memory), but you can feed it soft prompts | Q-Former bridge, frozen LLM | Only the bridge trains; finetuning is 188M parameters [arxiv:2305.06500 §3.5]. Watch the short-answer risk below [arxiv:2310.03744 §3.2] | | You want a cheap, reproducible baseline on public data | LLaVA-1.5 recipe | Two stages, about 26 hours of wall-clock time for 13B on 8 A100 GPUs [arxiv:2310.03744 §3.3] | | You need few-shot prompting over images, with no training | Emu2 pretrained base, with RICES-selected examples | Above Flamingo-80B and IDEFICS-80B on VQAv2 from 4 shots, with 37B parameters against 80B [arxiv:2312.13286 §3.1] [arxiv:2312.13286 tab. 1]. Below both at 0 shots [arxiv:2312.13286 tab. 1]. Results depend on example selection [arxiv:2312.13286 app. E.1] | | You need a large chat model and can clear the weights' license | Emu2-Chat recipe | Author-reported 84.9 on VQAv2 and 48.5 on MM-Vet [arxiv:2312.13286 tab. 2]. Chat uses 256 tokens per image [arxiv:2312.13286 §2.3.1]. Weights are listed as non-commercial [code: Emu2/README.md#L80-L82] | | One model must give short VQA answers and long chat answers | LLaVA-1.5 format prompts | The prompts teach when to be short; ablation gain on MME [arxiv:2310.03744 tab. 2] | | Your training data is small | Subsample the mixture | 50% keeps more than 98% of performance [arxiv:2310.03744 §5.1] | | You need fine text or OCR detail | HD tiling, after checking the cost | Small reported gains; more visual tokens [arxiv:2310.03744 tab. 3] | | Context or latency budget per image is tight | Q-Former bridge with a small K, or Shikra's 256 tokens | Q-Former output is K tokens (code default 32, paper silent) [arxiv:2305.06500 §2.3]. Shikra uses 256 tokens at 224 px [arxiv:2306.15195 §5.1]. LLaVA-1.5 uses 576 at 336 px [arxiv:2310.03744 §3.3] | | Inputs include several video frames, and the LLM must stay frozen | InstructBLIP-style frame concatenation, or Video-ChatGPT pooling | InstructBLIP encodes frames separately and concatenates them, so visual tokens grow with frame count [arxiv:2305.06500 §2.5]. Video-ChatGPT pools to T+N tokens and trains one linear layer [arxiv:2306.05424 §3.1] [arxiv:2306.05424 §3.2] | | You need a video chat model and can train only a projection | Video-ChatGPT recipe | Trainable part is one linear layer [arxiv:2306.05424 §3.2]; its gains are judged by GPT-3.5 and not by a human-rated benchmark [arxiv:2306.05424 §5.2] | | You need a video model that can be trained in more stages and has a stronger reported score | VideoChat2 recipe | Three stages, with the encoder unfrozen from stage 2 [arxiv:2311.17005 §4.2]. Author-reported 51.1 on MVBench [arxiv:2311.17005 tab. 2]; the paper's own ablation shows the gain depends on tuning the encoder and Q-Former [arxiv:2311.17005 §5.3] | | You need to test whether a video model uses time, not one frame | MVBench as the test | Multiple-choice items, 200 per task, scored without an LLM judge [arxiv:2311.17005 §1] [arxiv:2311.17005 §3.2]. Compare only rows with the same frame count and resolution [arxiv:2311.17005 tab. 2] | | You want an image-and-text chat model whose second stage is a light LoRA, and you can run or source a first stage yourself | mPLUG-Owl recipe | Stage 2 trains LoRA only, on text-only and image-text instructions [arxiv:2304.14178 §3.2]. The stage-1 numbers conflict and the stage-1 code is not released [arxiv:2304.14178 app. A] [code: mPLUG-Owl/pipeline/train.py]. Author-reported 66 of 82 OwlEval answers rated A or B against 54 for MiniGPT-4 [arxiv:2304.14178 §4.2] | | Your data has ordered images or video frames, and worked examples can be shown before each query | Otter recipe | Only the resampler and gated cross-attention train [arxiv:2305.03726 §3.1]. In the paper's single run, two-shot in-context training raises the two-shot MME-cognition score from 1310.7 to 1430.3 [arxiv:2305.03726 tab. 10]. The base is 9B parameters [arxiv:2305.03726 tab. 2] | | You want one model to accept several non-text inputs, and no benchmark score is needed yet | PandaGPT recipe | One frozen encoder covers six input types; only a projection and LoRA train, on image-text data [arxiv:2305.16355 §1] [arxiv:2305.16355 §3]. Evidence is example outputs only [arxiv:2305.16355 §4] | | The model must output boxes or points, with no new vocabulary | Shikra recipe | Coordinates are ordinary text [arxiv:2306.15195 §5.2]. Numbers beat added coordinate tokens in a toy test [arxiv:2306.15195 tab. 2] but cost more tokens per box [arxiv:2306.15195 §6.2]. The code is CC BY-NC 4.0, so reimplement from the paper for a commercial product | | Your images are aerial or satellite photos, and the model must answer about objects and boxes | GeoChat recipe | Task tokens pick the answer type and boxes are text [arxiv:2311.15826 §3.1]. The training set mixes three detection sets, one scene set and two VQA sets [arxiv:2311.15826 tab. 2]. Scene-classification gains are author-reported over LLaVA-1.5 [arxiv:2311.15826 tab. 5]. Images at 504 px give 1296 tokens [arxiv:2311.15826 §3.1] | | Users point at regions in an image, and coordinate text is awkward | ViP-LLaVA recipe | Marks are drawn on the pixels, so no region module is added [arxiv:2312.00784 §3.1]. Gain over coordinates is author-reported on PointQA [arxiv:2312.00784 app. Comparing Visual Prompts with Coordinates]. Full three-stage training is needed [arxiv:2312.00784 §5.1] | | The model must call external tools (detectors, OCR, generators) within a chat | LLaVA-Plus recipe | The model writes the tool call and reads the result [arxiv:2311.05437 §2.1]. Each call adds a second generation pass and the tool's latency; the paper does not measure latency [arxiv:2311.05437 §2.1] | | A planner must act on camera and LiDAR input, with decisions in a fixed list | DriveMLM-style decision output | Output maps onto a standard planner's decision states [arxiv:2312.09245 §3.2]. The data and weights are not public, so you must collect comparable data [arxiv:2312.09245 §4.1] | | You want a language model to describe and reason about people in a robot's view, and you can label your own scenes | Social-LLaVA recipe, LoRA on LLaVA-1.5 | Only LoRA adapters train, on about 2K scenes [arxiv:2501.09024 §III-B]. Scores are judged by people on 50 items, so treat them as a proof of concept [arxiv:2501.09024 tab. 1] [arxiv:2501.09024 §IV-B] | | You need answers about relative position and fit (left, right, above, whether an object fits), and you can generate data from 3D scans | RoboSpatial-style data, mixed with grounding question-answer pairs | Fine-tuning gains are author-reported on the authors' split [arxiv:2411.16537 tab. 3] [arxiv:2411.16537 tab. 11]. The official generator is non-commercial, so a commercial build must re-implement the pipeline from the paper [arxiv:2411.16537 Paper vs code]. The pipeline depends on GPT-4o to pick support objects, and its prompt is not given [arxiv:2411.16537 app. B.4] | | You want a commercial product | Train on your own data, not the released PandaGPT data or weights | Released data and delta weights are CC BY-NC 4.0 [code: README.md, "Usage and License Notices"]. This affects any recipe whose data you reuse. The Emu2 weights are listed as non-commercial too [code: Emu2/README.md#L80-L82] | ## Implementation sketch The common core: encode the image, map it to LLM tokens, and compute the loss on the response. Recipes differ in what trains and what the bridge reads. ```pseudocode # Shared training step; the vision encoder is frozen in the recipes that state it # [arxiv:2304.08485 §4.2] [arxiv:2305.06500 §2.2] [arxiv:2310.03744 §3.3] # [arxiv:2306.05424 §3.2] def training_step(image, instruction, response, vision, bridge, llm): # Patch or region features from the frozen encoder [arxiv:2310.03744 §3.3] feats = vision(image) # (N, d_v) # Bridge into the LLM embedding space. The InstructBLIP Q-Former also reads # the instruction; the LLaVA matrix, the MLP, MiniGPT-4's projection, # PandaGPT's projection, Video-ChatGPT's linear layer, Emu2's pooled # projection, mPLUG-Owl's abstractor and Otter's resampler ignore it # [arxiv:2305.06500 §2.3] [arxiv:2304.08485 §4.1] [arxiv:2304.10592 §3.1] # [arxiv:2306.05424 §3.1] [arxiv:2305.16355 §3] [arxiv:2312.13286 §2.1] # [arxiv:2304.14178 §3.1] [arxiv:2305.03726 §3.1] visual = bridge(feats, instruction) # (K, d_llm) # Loss on response tokens only [arxiv:2304.08485 §4.2] [arxiv:2305.06500 §2.2] # Train the bridge only (LLaVA stage 1, MiniGPT-4 stage 1, InstructBLIP, # Video-ChatGPT), bridge + LLM (LLaVA stage 2, LLaVA-1.5 stage 2) or # bridge + LoRA (PandaGPT) [arxiv:2306.05424 §3.2] [arxiv:2305.16355 §3] # [arxiv:2310.03744 §3.3] logits = llm(visual, instruction, response) return lm_loss(logits, response) ``` Practical notes from the cards: - Use the same response-format prompt at training and evaluation [arxiv:2310.03744 §3.2]. - The LLaVA-1.5 paper does not state its image preprocessing. The released code pads images to square with the CLIP mean colour in stage 2, and not in stage 1 [code: llava/train/train.py#L702-L716] [code: scripts/v1_5/finetune.sh#L15]. Inference should use the stage-2 preprocessing. - The original LLaVA paper gives stage 2 batch size 32. Its full-schedule script uses 16 per device, which is 128 on 8 GPUs with no gradient accumulation [arxiv:2304.08485 app. C] [code: scripts/finetune_full_schedule.sh#L31] [code: scripts/finetune_full_schedule.sh#L33]. Choose one and report it. - MiniGPT-4's paper gives 400 stage 2 steps. Its config runs 5 epochs of 200 iterations, which is 1,000 steps [arxiv:2304.10592 §3.3] [code: train_configs/minigpt4_stage2_finetune.yaml#L32-L33]. Its stage 1 caption end symbol and length cap are set in code and not in the paper [code: minigpt4/models/minigpt_base.py#L31]. - Shikra's coordinates are relative to a white square padded around the image, not to the original image [code: mllm/dataset/utils/transform.py#L52-L78]. The paper does not mention the padding, so a resized, non-square input will give different targets unless you pad the same way. - The InstructBLIP paper gives the sampling formula but not the manual weight values [arxiv:2305.06500 §2.4]. Its repository has no instruction-tuning training config, so the §2.6 settings cannot be checked against code [arxiv:2305.06500 §2.6]. - Video-ChatGPT's training and inference prompts differ in where the question sits relative to the video tokens. The paper's template puts the instruction first [arxiv:2306.05424 §3.2]. Inference puts the video tokens after the question [code: video_chatgpt/single_video_inference.py#L86]. Pick one placement and use it at both ends. - LLaVA-Plus's parser needs its literal field tags, `"thoughts🤔"`, `"actions🚀"` and `"value👉"`, one per line, with the action list as JSON [code: llava/mm_utils.py#L134-L146]. The paper names the fields but not the tags [arxiv:2311.05437 §2.1]. - ViP-LLaVA's prompt stroke widths scale with the image size against a 336-px anchor [code: llava/visual_prompt_generator.py#L303-L334]. The paper does not say so, and a resized input will change the mark thickness. - Emu2's chat wrapper sets `max_new_tokens` to 10 by default [code: Emu2/emu/chat.py#L47]. Longer answers are cut off unless the caller raises it, and the paper does not state an answer length [arxiv:2312.13286 app. E.2]. - GeoChat pads non-square images to a square before resizing during training, but its evaluation scripts resize and centre-crop with no padding. The two ends see different image content for non-square inputs. The paper does not mention this [code: geochat/train/train.py#L672-L690] [code: geochat/eval/batch_geochat_grounding.py#L86]. - The Emu2 code pads and truncates on the left [code: Emu2/emu/emu.py#L57-L58]. Batched inputs must follow the same convention. - mPLUG-Owl's training transform includes a random horizontal flip, which can turn a left-right answer into the wrong one. The paper mentions only the resize [arxiv:2304.14178 §4.1] [code: mPLUG-Owl/pipeline/data_utils/processors/caption_processor.py#L27-L28]. - The mPLUG-Owl stage-2 script sets a global batch of 256, but the value never reaches the trainer. With eight processes and micro-batch 4, the effective batch is 32 [arxiv:2304.14178 §4.1] [code: mPLUG-Owl/scripts/train_it.sh#L17-L31]. - Otter's table gives six epochs, and the repository's scripts use three. Check which one a run uses before comparing numbers [arxiv:2305.03726 tab. 11] [code: README.md#L172]. ## Contradictions **Why InstructBLIP overfits to short answers.** LLaVA-1.5 reports that InstructBLIP gives short answers even when a detailed reply is requested. It attributes this to ambiguous prompt formats and to tuning only the Q-Former [arxiv:2310.03744 §3.1] [arxiv:2310.03744 §3.2]. InstructBLIP's own ablations test instruction-aware features and data balancing. They do not test prompt format or LLM tuning [arxiv:2305.06500 tab. 2]. Neither side resolves the question. **ScienceQA numbers for the same model.** LLaVA-1.5's table lists InstructBLIP (Vicuna-7B) at 60.5 on SciQA-IMG [arxiv:2310.03744 tab. 3]. That matches InstructBLIP's zero-shot row [arxiv:2305.06500 tab. 1]. InstructBLIP's finetuned row gives 79.5 for the same model [arxiv:2305.06500 tab. 3]. These are different protocols and are not comparable. **Hyperparameters and code.** The LLaVA-1.5 hyperparameter table matches the released v1.5 scripts [arxiv:2310.03744 tab. 9] [code: scripts/v1_5/finetune.sh#L3-L37]. The HD variant is not reproduced by any v1.5 script. Its tile size, padding and feature order differ from the paper's description [code: llava/mm_utils.py#L12-L39]. **Which parameters train in Video-ChatGPT.** The paper says only the linear layer is updated [arxiv:2306.05424 §3.2]. Its released training flags also mark the whole input embedding matrix as trainable [code: video_chatgpt/model/video_chatgpt.py#L278-L282] [code: docs/train_video_chatgpt.md#L68-L69]. The card reads the code and concludes that the gradient reaches only the boundary-token rows. That reading was not run, so check it before relying on it. **Visual7W for the same ViP-LLaVA model.** The main table gives 87.91% for ViP-LLaVA-13B [arxiv:2312.00784 §5.2]. The appendix table on GPT-4V data gives 88.28 for the 13B model [arxiv:2312.00784 app. Effect of Optional GPT-4V Region-Level Instruction Data]. The abstract uses the main table. Both numbers are reported and neither is resolved here. **Whether RSVQA-LRBEN is zero-shot for GeoChat.** Within the same paper, Table 1 lists 56k LRBEN training pairs, but Table 6 calls the model "not specifically finetuned on target dataset" [arxiv:2311.15826 tab. 1] [arxiv:2311.15826 tab. 6]. The paper does not explain the difference. Neither side is resolved here, so treat the LRBEN numbers as in-distribution. **Which LiDAR encoder DriveMLM uses.** The method section says the LiDAR encoder is randomly initialised [arxiv:2312.09245 §3.3]. The experiments section says a GD-MAE encoder fine-tuned on ONCE is used [arxiv:2312.09245 §4.2]. The paper does not say which one the reported model uses. The ablation finds that point clouds do not help [arxiv:2312.09245 tab. 5]. **Whether LLM judges are acceptable for video.** MVBench argues that scoring open answers with an LLM adds bias, and uses multiple-choice items for that reason [arxiv:2311.17005 §1] [arxiv:2311.17005 §3.2]. Its own conversation scores are ChatGPT-judged [arxiv:2311.17005 tab. 3]. Video-ChatGPT's main quality results use a GPT-3.5 judge [arxiv:2306.05424 §5.2]. The cards do not settle which measure is the better one, so compare judged and multiple-choice numbers apart. **LLaVA-1.5-13B on MM-Vet.** LLaVA-1.5 reports 36.1 for the 13B model at 336 px [arxiv:2310.03744 tab. 4]. The Emu2 comparison table lists 35.4 for LLaVA-1.5-13B [arxiv:2312.13286 tab. 2]. The cards do not say why the numbers differ. Both are reported and neither is resolved here. **Which LoRA targets mPLUG-Owl trains.** The abstract and §1 describe LoRA on the LLM and the abstractor together [arxiv:2304.14178 §1]. §3.2 freezes the abstractor and puts LoRA on the LLM only [arxiv:2304.14178 §3.2]. The released code follows §3.2 and attaches LoRA to the language model's attention projections [code: mPLUG-Owl/pipeline/train.py#L143-L154]. Neither side is resolved here. **mPLUG-Owl stage-1 values.** The text gives learning rate 1e-4, 2,000 warm-up steps, weight decay 0.1 and a batch of 2.1M tokens [arxiv:2304.14178 §4.1]. The appendix table gives 2e-4, 375 steps, 0.01 and 4,096 image-caption pairs [arxiv:2304.14178 app. A]. The stage-1 code is not in the repository, so the conflict cannot be settled from code. **Whether RoboSpatial hurts general benchmarks.** The paper's text says trained models keep or slightly improve performance on every general benchmark [arxiv:2411.16537 app. D.3]. Its table shows MME cognition for LLaVA-NeXT falling from 305.4 to 293.2 [arxiv:2411.16537 tab. 10]. Neither side is resolved here, so measure the general score for your own model. **Which parts of Otter train.** The paper says only the perceiver resampler and the gated cross-attention train [arxiv:2305.03726 §3.1]. The released code also unfreezes the input embeddings, and the LM head for LLaMA-style decoders [code: src/otter_ai/models/otter/modeling_otter.py#L648]. The paper also gives 1.4B trainable parameters in the text and 1.3B in Table 2 [arxiv:2305.03726 §3.1] [arxiv:2305.03726 tab. 2]. Neither side is resolved here, so budget memory for the larger set. ## Papers - [Visual Instruction Tuning](/papers/2304.08485): frozen CLIP, one linear matrix, two stages, 158K GPT-4-generated samples; GPT-4-judged benchmark. - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](/papers/2304.10592): frozen BLIP-2 encoder and Q-Former with one projection; a curated second stage of about 3,500 descriptions. - [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](/papers/2305.06500): Q-Former that reads the instruction, frozen LLM, 13 held-in and 13 held-out datasets. - [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](/papers/2306.05424): frame-pooled CLIP tokens, one trainable linear layer, 100K video-instruction pairs. - [Improved Baselines with Visual Instruction Tuning](/papers/2310.03744): LLaVA-1.5, an MLP bridge with full LLM tuning, response-format prompts, and a high-resolution variant. - [PandaGPT: One Model To Instruction-Follow Them All](/papers/2305.16355): ImageBind encoder shared across six input types, a linear projection and LoRA, 160k image-instruction examples; examples only, no benchmarks. - [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](/papers/2304.14178): learnable visual abstractor on LLaMA-7B; encoder trained in stage 1, LoRA-only stage 2; author-reported OwlEval ratings. - [Otter: A Multi-Modal Model with In-Context Instruction Tuning](/papers/2305.03726): OpenFlamingo-9B with in-context image and video examples from MIMIC-IT; author-reported benchmarks and in-context gains. - [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](/papers/2306.15195): coordinates written as text, one linear layer, 5,922 GPT-4 referential dialogues; author-reported referring and PointQA results. - [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](/papers/2312.00784): marks drawn on the image, LLaVA-style model, 520k region-prompt examples; author-reported gains on region benchmarks. - [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](/papers/2312.09245): fine-tuned LLM that outputs speed and path decisions for a motion planner; closed-loop results in CARLA, data not public. - [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](/papers/2311.05437): tool-call dialogues in visual instruction tuning, with thoughts, actions and value fields; author-reported benchmark gains. - [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](/papers/2311.17005): 20 temporal video tasks as multiple-choice items; also describes the VideoChat2 three-stage training. - [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](/papers/2311.15826): LLaVA-1.5 layout with 504 px input, `[grounding]`, `[identify]` and `[refer]` task tokens, and LoRA on the LLM; author-reported remote-sensing scene and grounding results. - [Generative Multimodal Models are In-Context Learners](/papers/2312.13286): Emu2, a 37B interleaved model with pooled image embeddings; chat tuning at 256 tokens per image; author-reported VQAv2 and MM-Vet; non-commercial weights. - [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](/papers/2411.16537): about 3M generated spatial questions from 3D scans used to fine-tune 2D and 3D VLMs; author-reported held-out and robot gains. - [Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces](/papers/2501.09024): LLaVA-1.5-7B with LoRA on about 40K human-annotated social-navigation VQA pairs; author-reported human ratings above prompted GPT-4V and Gemini 1.5 Pro. - [A Survey on Multimodal Large Language Models](/papers/2306.13549): the encoder, connector and LLM split and the three training stages; no experiments of its own.