--- kind: concept slug: multimodal-language-model title: Multimodal language model summary: >- A language model that reads images, video, audio or other signals next to text and answers in text. Designs differ in how the non-text input enters the model, which parts train, and whether the output can also be masks or boxes: a frozen LLM behind a small bridge, a natively multimodal decoder, a fully fine-tuned generalist, or one token vocabulary for everything. tags: [multimodal, vision-language, video, audio, medical-ai, architecture, instruction-tuning, encoder-decoder, segmentation, safety, inference, post-training, speech, continual-learning, hallucination, captioning, ocr, grounding, in-context-learning, image-generation, motion-generation, spatial-reasoning, vision-encoder, benchmark, remote-sensing, multimodal-embedding, navigation, failure-correction, zero-shot, driving, synthetic-data, social-navigation, long-context, region-grounding, robot-manipulation] papers: ["2111.02114", "2111.09734", "2202.03052", "2206.08916", "2209.06794", "2209.09513", "2301.12597", "2302.00923", "2302.11713", "2304.03086", "2304.08485", "2304.10592", "2305.06500", "2305.07895", "2305.11000", "2305.16355", "2306.05424", "2306.13549", "2306.15195", "2307.14334", "2308.09126", "2308.10741", "2311.03356", "2311.05437", "2311.06242", "2311.16922", "2311.17005", "2311.17600", "2312.00784", "2312.02228", "2312.09245", "2312.11805", "2312.13286", "2312.16217", "2403.05530", "2403.06764", "2404.14219", "2410.21276", "2503.01785", "2503.10265", "2504.06958", "2504.17432", "2508.07173", "2601.14724", "2602.01990", "2604.05081", "2606.00959", "2304.14178", "2305.03726", "2305.11175", "2306.14795", "2401.06209", "2412.14171", "2311.15826", "2510.05014", "2509.04018", "2512.08186", "2411.05755", "2411.11394", "2411.14256", "2501.09024", "2404.18416", "2406.01584", "2411.16537"] related: [vision-transformer, multi-query-attention, knowledge-distillation, chain-of-thought-prompting, image-captioning, dataset-curation, multi-task-learning, object-detection, instruction-tuning, visual-instruction-tuning, vision-language-pretraining, low-rank-adaptation, self-attention, bipartite-matching, vector-quantization, self-supervised-learning, self-consistency, visual-question-answering-evaluation, segmentation-evaluation, policy-gradient, rlhf, llm-evaluation, adversarial-evaluation, adversarial-robustness, red-teaming, llm-as-a-judge, benchmark-validity, multi-agent-systems, retrieval-augmented-generation, dense-retrieval, contrastive-learning, masked-image-modeling, tool-use, kv-cache-management, token-pruning, positional-encoding, learning-from-demonstration, sim-to-real-transfer, importance-weighting, vision-language-action-model, hallucination-mitigation, in-context-learning, language-model-pretraining, preference-optimization, constrained-decoding, decoding-strategies, discrete-speech-units, mixture-of-experts, continual-learning, optical-character-recognition, visual-grounding, referring-segmentation, text-to-image-generation, flow-matching, diffusion-models, action-chunking, visual-prompting, zero-shot-learning, imitation-learning, data-augmentation, human-evaluation, self-training, uncertainty-estimation, monocular-depth-estimation, 3d-scene-graph] status: draft updated_at: 2026-11-01 --- ## Overview A multimodal language model takes a non-text input, such as an image, a video, an audio clip or a depth map, next to text, and produces an answer. Three choices separate the designs. 1. **How the non-text input reaches the model.** Through a small trained bridge into a frozen LLM, through a shared embedding space from a frozen encoder, through the same token vocabulary as text, or through one network trained from the start on all modalities. 2. **Which parts train.** Only the bridge, the bridge plus adapters, or every weight. 3. **What the output is.** Text only, or text plus structured outputs such as boxes, masks and depth, written as tokens or produced by an extra decoder. A survey of the field describes the same parts as a modality encoder, a pre-trained LLM and a connector between them [arxiv:2306.13549 §2]. Most designs below differ only in what the connector is and which parts it lets train. The bridge designs are the most common in the cards. InstructBLIP keeps both its image encoder and its LLM frozen and trains only a Q-Former [arxiv:2305.06500 §2.2] [arxiv:2305.06500 §2.3]. Video-ChatGPT and PandaGPT also keep the LLM frozen. Each trains a linear map into the LLM's input space, and PandaGPT adds LoRA adapters [arxiv:2306.05424 §3.1] [arxiv:2305.16355 §3]. At the other end, Med-PaLM M updates every weight of a multimodal base for one domain [arxiv:2307.14334 §4.2]. Gemini 1.0 is trained as a multimodal decoder from the start, and the report does not describe its visual encoder or bridge [arxiv:2312.11805 §2]. The same family also writes tool calls, produces embeddings, picks robot or driving actions, and is tested for safety and for which input it relies on. Those uses have their own sections below. ## Variants ### Frozen LLM with an instruction-aware bridge Used by [InstructBLIP](/papers/2305.06500). The image encoder is a frozen ViT-g/14, and the LLM is a frozen FlanT5 or Vicuna model [arxiv:2305.06500 §2.6]. The Q-Former's learned queries attend to the image features, and the instruction tokens also go into the Q-Former. That makes the visual features depend on the task being asked [arxiv:2305.06500 §2.3]. Removing this instruction-aware step lowers the held-in average for FlanT5-XL from 94.1 to 89.8 [arxiv:2305.06500 tab. 2]. For video, the paper samples four frames, runs each through the encoder and Q-Former separately, and concatenates the outputs, so the LLM receives four times as many visual vectors [arxiv:2305.06500 §2.5]. Only the Q-Former is trained. The paper reports 188M trainable parameters in finetuning, against 1.2B for BLIP-2 finetuning [arxiv:2305.06500 §3.5]. ### Frozen LLM with a linear bridge over video Used by [Video-ChatGPT](/papers/2306.05424). A frozen CLIP ViT-L/14 encodes each frame, and the patch features are pooled two ways: averaged over the grid to give one vector per frame, and averaged over frames to give one vector per patch position. The two sets are stacked, giving T+N tokens, and one linear layer maps them into the 7B Vicuna model's embedding space [arxiv:2306.05424 §3.1] [arxiv:2306.05424 eq. 1] [arxiv:2306.05424 eq. 2]. The bridge gets no instruction input, so the visual tokens do not depend on the question. The paper does not test whether that matters. Only the linear layer is trained, and the paper reports about 3 hours on 8 A100 40GB GPUs [arxiv:2306.05424 §5.1]. The paper states no frame count. Its released code samples up to 100 frames at fixed positions, so frame handling has to come from the code if you want to match the released weights (see the [card's Paper vs code](/papers/2306.05424)). ### Frozen LLM with a query bridge and no instructions Used by [BLIP-2](/papers/2301.12597). A frozen image encoder and a frozen LLM are joined by a Querying Transformer with 32 learnable queries of width 768, initialised from BERT-base [arxiv:2301.12597 §3.1]. The Q-Former is the only part that is trained in the first stage. It learns three losses at once: image-text contrastive, image-grounded text generation and image-text matching, each with its own attention mask [arxiv:2301.12597 §3.2]. In the second stage, a fully connected layer maps the query outputs into the LLM's embedding width, and the result is prepended to the text as soft visual prompts [arxiv:2301.12597 §3.3]. The paper reports 188M trainable parameters for the Q-Former [arxiv:2301.12597 tab. 1], but other tables give 103M to 108M for the same component [arxiv:2301.12597 tab. 2]. The paper says the first stage matters: without it, zero-shot VQA is much lower for both LLM types [arxiv:2301.12597 §4.1]. Zero-shot VQAv2 test-dev reaches 65.0 with a ViT-g encoder and FlanT5-XXL, against 56.3 for Flamingo-80B [arxiv:2301.12597 tab. 2]. The paper's "8.7%" is a difference of percentage points, not a relative gain [arxiv:2301.12597 §4.1]. The paper's zero-shot results are for this design alone; instruction tuning of the bridge is the InstructBLIP variant above. ### Frozen encoder, one linear projection, two-stage tuning Used by [LLaVA](/papers/2304.08485) and [MiniGPT-4](/papers/2304.10592). LLaVA takes features from a frozen CLIP ViT-L/14, uses the layer before the last, and maps them through one trainable matrix into Vicuna [arxiv:2304.08485 §4.1] [arxiv:2304.08485 tab. 8]. Stage 1 trains only that matrix on about 595K caption pairs. Stage 2 trains the matrix and the LLM on 158K GPT-4-written instruction samples, with the image encoder still frozen [arxiv:2304.08485 §4.2]. The authors report 85.1 on their COCO benchmark, scored by a text-only GPT-4 judge against a reference answer [arxiv:2304.08485 tab. 4]. On ScienceQA, LLaVA reaches 90.92 on the test split [arxiv:2304.08485 tab. 7], and the authors list "MM-CoT Large" at 91.68, a number that differs from the one in its own paper (see Contradictions). MiniGPT-4 freezes a BLIP-2 vision stack and Vicuna, and trains one linear layer [arxiv:2304.10592 §3]. Stage 1 aligns that layer on about 5M image-caption pairs. The authors say caption data alone gives unnatural output, with repeated words and fragments [arxiv:2304.10592 §3.1]. Stage 2 fine-tunes on about 3,500 detailed descriptions that were produced by the stage-1 model and then edited by ChatGPT and by hand [arxiv:2304.10592 §3.2]. The paper reports 400 stage-2 steps at batch 12 [arxiv:2304.10592 §3.3], but the released config runs more steps (see the [card's Paper vs code](/papers/2304.10592)). On a 100-image set, human judges rate 65 of 100 requests satisfied against 5 for BLIP-2, and the failure rate of detailed descriptions falls from 35% to 2% after stage 2 [arxiv:2304.10592 tab. 1] [arxiv:2304.10592 tab. 3]. The paper puts stage 1 at about ten hours on four A100 GPUs and stage 2 at about seven minutes on one [arxiv:2304.10592 §3.1] [arxiv:2304.10592 §3.3]. ### Learned visual abstractor, then LoRA on a language model Used by [mPLUG-Owl](/papers/2304.14178). A CLIP ViT-L/14 encoder feeds a learnable visual abstractor, which compresses the patch features into a short set of visual tokens for LLaMA-7B [arxiv:2304.14178 §3.1]. The paper does not state how many visual tokens the abstractor produces. Stage 1 trains the encoder and the abstractor on image-caption pairs while the LLM stays frozen, with the loss on text tokens only [arxiv:2304.14178 §3.2]. Stage 2 freezes the whole model and trains LoRA adapters on the LLM, using text-only and image-text instructions [arxiv:2304.14178 §3.2]. The authors report 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4. The rating is manual, and the paper does not say how many raters scored the answers [arxiv:2304.14178 §4.2]. The stage-1 settings conflict inside the paper. The text gives learning rate 0.0001 and 2,000 warmup steps [arxiv:2304.14178 §4.1], and the appendix table gives 2e-4 and 375 steps [arxiv:2304.14178 app. A]. The released repository contains only the stage-2 code, so stage 1 cannot be checked (see the [card's Pitfalls](/papers/2304.14178)). ### Frozen LLM with in-context examples and gated cross-attention Used by [Otter](/papers/2305.03726). The base is OpenFlamingo-9B, with a frozen CLIP ViT-L/14 encoder and a frozen MPT-7B decoder. A perceiver resampler and gated cross-attention layers inside the decoder are the only trained parts [arxiv:2305.03726 §3.1]. The paper puts this at about 1.4B parameters [arxiv:2305.03726 §3.1]. The difference from the bridges above is the training data. Each MIMIC-IT item holds a query and a set of in-context examples, which are other image-instruction-response sets shown before the query [arxiv:2305.03726 §4]. For the LLaVA-derived subsets, the examples are the most similar training items by CLIP image similarity or instruction text similarity [arxiv:2305.03726 §4.1]. The MIMIC-IT dataset has about 3M instances and 2.2M unique instructions [arxiv:2305.03726 tab. 1]. The paper reports that training with in-context examples converges faster and does better at few-shot evaluation. On MME cognition with two-shot evaluation, a model trained without them scores 1,310.7 and one trained with two-shot examples scores 1,430.3 [arxiv:2305.03726 tab. 10]. The runs are single, and the paper reports no variance. The video results use a ChatGPT-based judge [arxiv:2305.03726 §6.2]. ### Frozen CLIP prefix and a small mapper for captions Used by [ClipCap](/papers/2111.09734). A frozen CLIP image encoder gives one embedding, and a mapping network turns it into k vectors that sit in front of the caption tokens of GPT-2 [arxiv:2111.09734 §3.1] [arxiv:2111.09734 eq. 3] [arxiv:2111.09734 eq. 4]. The method needs only images and captions [arxiv:2111.09734 §1]. The paper has two variants. One fine-tunes GPT-2 with a single-hidden-layer MLP mapper, 156M parameters in total. The other freezes GPT-2 and trains a transformer mapper, 43M parameters [arxiv:2111.09734 §3.2] [arxiv:2111.09734 tab. 1]. On Conceptual Captions, the MLP variant reaches 87.26 CIDEr against 77.57 for VLP, and the transformer variant reaches 71.82 [arxiv:2111.09734 tab. 1 (A)]. On COCO, the transformer variant reaches 113.08 CIDEr against 124.12 for Oscar, which also uses object tags [arxiv:2111.09734 tab. 1 (C)] [arxiv:2111.09734 §4]. The transformer variant trains for 6 hours on one GTX 1080, with the prefix set to 10 vectors [arxiv:2111.09734 tab. 1] [arxiv:2111.09734 §4]. The released README uses a prefix of 40 for that variant (see the [card's Paper vs code](/papers/2111.09734)). ### One embedding space for many modalities Used by [PandaGPT](/papers/2305.16355). A frozen ImageBind encoder maps image, video, audio, depth, thermal and IMU inputs into one embedding space. One trainable linear projection maps the resulting vector into a frozen Vicuna-13B, and LoRA adapters on the attention modules are the only other trainable weights [arxiv:2305.16355 §3]. Training uses 160k image-language examples for two epochs [arxiv:2305.16355 §3]. Other modalities work without their own training because the encoder already places them in the shared space [arxiv:2305.16355 §1]. Several inputs are combined by adding their vectors [arxiv:2305.16355 §4]. The paper reports example conversations and no benchmark numbers, so its cross-modal results are demonstrations, not measurements [arxiv:2305.16355 §4]. The released data and delta weights carry non-commercial terms, so a commercial build cannot start from them (see the [card's Pitfalls](/papers/2305.16355)). ### Natively multimodal decoder Used by [Gemini 1.0](/papers/2312.11805). A decoder-only Transformer with a 32,768-token context, trained on text, images, audio and video [arxiv:2312.11805 §2] [arxiv:2312.11805 tab. 1]. Audio enters as 16 kHz Universal Speech Model features rather than as a transcript [arxiv:2312.11805 §2]. Video enters as frames in the context; the video benchmarks use 16 frames per clip [arxiv:2312.11805 §5.2.2]. The model can also emit images as discrete tokens, though the image decoder is not described [arxiv:2312.11805 §2]. The family has four sizes: Ultra and Pro, whose parameter counts are not stated, and Nano-1 (1.8B) and Nano-2 (3.25B), which are distilled from larger Gemini models and 4-bit quantized [arxiv:2312.11805 tab. 1]. The quantization method and distillation objective are not stated [arxiv:2312.11805 §2]. Post-training follows a pipeline of prompts, supervised fine-tuning, a reward model trained on human preferences, and iterated RLHF [arxiv:2312.11805 §6.3]. The report gives no RL algorithm, KL penalty, learning rate or data sizes for it [arxiv:2312.11805 §6.3]. ### Sparse mixture-of-experts decoder for million-token context Used by [Gemini 1.5](/papers/2403.05530). Pro is a sparse mixture-of-experts Transformer: a learned router sends each input to part of the parameters, so the total can grow while the work per input stays fixed [arxiv:2403.05530 §3.1]. The authors say architecture changes let inputs reach 10 million tokens without losing quality, but do not describe those changes [arxiv:2403.05530 §3.1]. Flash is a dense Transformer with the same interface. It is distilled online from Pro, and its optimiser is named only as higher-order and preconditioned [arxiv:2403.05530 §3.2]. The parameter count, the training data mixture beyond its domains and the training hyperparameters are not stated [arxiv:2403.05530 §4] [arxiv:2403.05530 app.]. ### Fully fine-tuned generalist from a multimodal base Used by [Med-PaLM M](/papers/2307.14334), built on PaLM-E at 12B, 84B and 562B parameters. PaLM-E's language model is a decoder-only Transformer, and its image encoder is a ViT with 4B parameters for the 12B model and 22B for the 84B and 562B models [arxiv:2307.14334 §4.1]. Each 224 x 224 image becomes 256 tokens [arxiv:2307.14334 app. A.2.2], and each fine-tuning input holds at most one image [arxiv:2307.14334 §4.2]. Unlike InstructBLIP, nothing is frozen. All parameters are updated on MultiMedBench, with instruction-style prompts and text targets [arxiv:2307.14334 §4.2]. Learning rate is 5e-5 for the 12B and 84B models and 2.5e-5 for the 562B model, with a constant schedule and Adafactor [arxiv:2307.14334 tab. A.2] [arxiv:2307.14334 §4.2]. For most tasks, the exemplar's image is replaced by a text placeholder, so the prompt is text-only except for the final image [arxiv:2307.14334 §4.2] [arxiv:2307.14334 fig. 2]. ### Gemini tuned for medicine, with self-trained search and long context Used by [Med-Gemini](/papers/2404.18416). The paper starts from Gemini 1.0 and 1.5 models and builds three tracks, each with its own tuned model. The text track, Med-Gemini-L 1.0 on Gemini 1.0 Ultra, is fine-tuned on reasoning chains the model writes itself. Each training question gets one chain without search results and one with them, and chains that end in a wrong answer are dropped [arxiv:2404.18416 §2.1]. At inference the model samples several reasoning paths and searches the web only when those paths disagree strongly, judged by the entropy of their answers. The entropy threshold is not stated [arxiv:2404.18416 §2.1]. The image track, Med-Gemini-M 1.5 on Gemini 1.5 Pro, is instruction-tuned on eight medical multimodal tasks from six datasets. Each dataset's share of the mixture is roughly proportional to its size [arxiv:2404.18416 §2.2] [app. D.1]. Med-Gemini-S 1.0 adds a cross-attention encoder for raw 12-lead ECG signals to Gemini 1.0 Nano [arxiv:2404.18416 §2.2]. The long-context track uses no new weights: a patient's notes go into one prompt for a two-step check, and a video goes in with a task instruction and no examples [arxiv:2404.18416 §2.3]. The results are author-reported. On MedQA the text model reaches 91.1 against 90.2 for GPT-4 with MedPrompt [arxiv:2404.18416 tab. 1]. On ECG-QA the Nano-based model reaches 57.7 against 51.6 [arxiv:2404.18416 tab. 2]. The results are not uniform: the EHR search F1 is 0.77 against 0.78 for a heuristic baseline [arxiv:2404.18416 tab. 3], and surgical view classification is 55.2 against 67.0 for a ResNet3D the authors reproduced [arxiv:2404.18416 tab. 3]. The multimodal baselines are not the same in every row [arxiv:2404.18416 tab. 2]. The model weights and most code are not released, and the training recipe leaves out learning rates and step counts (see the [card's Paper vs code](/papers/2404.18416)). Used by [Unified-IO](/papers/2206.08916). Text is written with SentencePiece tokens, points and boxes with 1000 location tokens, and images or dense maps such as depth with the codes of a 16384-entry VQ-GAN trained on ImageNet [arxiv:2206.08916 §3.1] [arxiv:2206.08916 §3.2]. One T5-style encoder-decoder reads the prompt and the image patches and writes the answer as tokens, which are then converted back into a box, a mask, a depth map or a sentence [arxiv:2206.08916 §3.1]. The model is trained on 95 datasets with no task-specific heads [arxiv:2206.08916 §3.3]. Model sizes run from 71M to 2,925M parameters [arxiv:2206.08916 tab. 2]. The XL model reports 64.3 on the GRIT test split against 32.0 for GPV-2 [arxiv:2206.08916 tab. 3]. The authors say the ImageNet-only image tokenizer limits image generation and depth and normal quality [arxiv:2206.08916 §4.6]. The released code is inference-only, so the training recipe rests on the paper alone (see the [card's Paper vs code](/papers/2206.08916)). [OFA](/papers/2202.03052) uses the same idea with one shared vocabulary: BPE text, 1,000 location tokens for box corners, and VQGAN-style image codes, so a 256 x 256 image becomes a 16 x 16 code sequence [arxiv:2202.03052 §3.1]. Every task, pretraining included, is generation from an instruction plus inputs [arxiv:2202.03052 §3.2]. The authors pretrain on public image-text pairs plus unimodal data; the abstract says 20M pairs, while the data table lists 15.25M captioning and matching samples [arxiv:2202.03052 tab. 11]. Reported results include 82.0 on VQA test-std and 154.9 CIDEr for COCO captions with CIDEr optimisation [arxiv:2202.03052 tab. 2] [arxiv:2202.03052 tab. 3]. The unimodal results are not always the best, and the paper's zero-shot results are sensitive to the wording of the instruction [arxiv:2202.03052 §4.3]. ### Dental image, voice and records in one LLM, described but not built Proposed in a review, not built or measured. The authors sketch a system in which three inputs, a dental X-ray or cone-beam CT, a patient's voice and old records, feed one LLM that proposes a diagnosis and a treatment [arxiv:2304.03086 fig. 11]. The voice is converted to text and plotted as a waveform and spectrogram, and the results go to the same model [arxiv:2304.03086 p. 9]. The paper names no model for the image step, gives no training data and reports no evaluation. Each example is one output the authors chose [arxiv:2304.03086 fig. 1-12]. Use it as a list of possible inputs and risks, not as a design. Its own caries example misses a finding the authors note themselves [arxiv:2304.03086 p. 10]. Card: [ChatGPT for Shaping the Future of Dentistry](/papers/2304.03086). ### Vision-centric outputs written as class and location tokens Used by [VisionLLM](/papers/2305.11175). A language-guided tokenizer reads the image at four scales, cross-attends to BERT features of the prompt, and turns them into M = 100 learned image tokens with a Deformable-DETR-style transformer [arxiv:2305.11175 §3.3]. An Alpaca-7B decoder with LoRA then writes the answer. The vocabulary gains 1,025 location tokens, covering offsets of -512 to 512 from a reference point, and 512 class tokens [arxiv:2305.11175 §3.4]. The instruction names the output format, and the decoder reads those format queries first, so the outputs come out in parallel rather than one token at a time [arxiv:2305.11175 §3.4]. Object detection and instance segmentation use the same output. A box is two corner points, and a mask is a polygon of boundary points: more than eight in the method, and 8 to 24 in the experiments [arxiv:2305.11175 §3.2] [arxiv:2305.11175 §4.2]. Training uses a focal loss on the tokenizer queries and bipartite matching for set outputs [arxiv:2305.11175 app. B]. Its reported box AP is 44.6 with ResNet-50 and 60.2 with InternImage-H [arxiv:2305.11175 tab. 1]. The number of location bins matters: 257 bins give 34.9 AP and 1,025 give 44.8 in the ablation [arxiv:2305.11175 tab. 3c]. The linked repository has no implementation of this method at the pinned commit (see the [card's Paper vs code](/papers/2305.11175)), so an implementation rests on the paper alone. ### Continuous image embeddings in one autoregressive stream Used by [Emu2](/papers/2312.13286). Each image becomes 64 embeddings: patch features from a visual encoder are average-pooled to an 8 x 8 grid, then mapped by one linear projection [arxiv:2312.13286 §2.1]. The embeddings sit between image markers in an interleaved sequence with text. One autoregressive model of 37B parameters predicts the next text token, with cross-entropy, or the next image embedding, with a regression loss. The paper gives no weighting between the two terms [arxiv:2312.13286 §2.2.2]. Because image outputs are continuous vectors rather than vocabulary entries, a separate SDXL-based diffusion decoder turns predicted embeddings into pictures [arxiv:2312.13286 §2.2.3]. The language-model pretraining runs in two stages, the second with the visual encoder frozen [arxiv:2312.13286 §2.2.2]. The few-shot results rise with the number of demonstrations. VQAv2 goes from 33.5 at zero shots to 67.0 at four and 68.8 at sixteen [arxiv:2312.13286 tab. 1]. The instruction-tuned Emu2-Chat reaches 84.9 on VQAv2 [arxiv:2312.13286 tab. 2]. The zero-shot setting needs a particular prompt: the plain one made the model write sentences, so the authors used a modified one [arxiv:2312.13286 app. E.1]. The released code is inference-only, and its image decoding runs the full model once per output slot (see the [card's Pitfalls](/papers/2312.13286)). ### Discrete speech units as vocabulary tokens Used by [SpeechGPT](/papers/2305.11000). A self-supervised speech encoder gives frame features, k-means assigns each frame to a cluster, and adjacent repeats are removed to give a unit sequence [arxiv:2305.11000 §4.1]. The paper does not state the number of clusters. The unit indices are added to the LLM's vocabulary, so speech in and speech out are ordinary tokens, with no recogniser or synthesiser in the loop [arxiv:2305.11000 §4.2] [arxiv:2305.11000 §1]. Removing repeats also removes durations, so the model does not control timing or emphasis [arxiv:2305.11000 §3.1]. Training has three stages [arxiv:2305.11000 §4.2]. Unit pre-training uses about 60K hours of English audiobook speech [arxiv:2305.11000 §5.1]. Instruction tuning uses about 9 million speech-text pairs from English ASR corpora, turned into recognition and synthesis tasks [arxiv:2305.11000 §3.1]. A LoRA stage then tunes on chain-of-modality data, where a text answer is followed by a speech answer, with rank 8 and alpha 16 [arxiv:2305.11000 §3.2] [arxiv:2305.11000 app. D]. The paper's backbone is LLaMA-13B, while the released scripts use 7B (see the [card's Paper vs code](/papers/2305.11000)). The evidence is qualitative: a few transcription and read-aloud examples, and ten spoken dialogues judged by people. No metric and no baseline are reported [arxiv:2305.11000 §5.2] [arxiv:2305.11000 tab. 1] [arxiv:2305.11000 tab. 2]. The paper lists three limits: no control over emotion or tone, a text answer written before the speech answer, and no multi-turn support [arxiv:2305.11000 §6]. ### Human motion codes as vocabulary tokens Used by [MotionGPT](/papers/2306.14795). The same idea as speech units, applied to 3D human motion rather than audio. A VQ-VAE turns a motion clip into codes from a codebook of 512 entries, after a temporal downsampling of four [arxiv:2306.14795 §3.1] [arxiv:2306.14795 §4.1]. The codes become words in a Flan-T5 vocabulary, wrapped in start and end tokens [arxiv:2306.14795 §3.2]. The base model is Flan-T5 at 220M parameters [arxiv:2306.14795 §4.1]. Training has three stages. The tokenizer is trained first and then frozen. The language model is pretrained on span corruption over text and motion, and on supervised motion-text pairs from HumanML3D and KIT [arxiv:2306.14795 §3.3]. It is then instruction-tuned on prompt templates, with more than a thousand prompts across about 15 tasks [arxiv:2306.14795 §3.3] [arxiv:2306.14795 tab. 13]. One model covers text-to-motion, motion-to-text, prediction and in-between completion [arxiv:2306.14795 §4]. On HumanML3D, the task-tuned model reaches FID 0.232 for text-to-motion, while T2M-GPT reports 0.116 [arxiv:2306.14795 tab. 3]. For motion-to-text, BLEU@4 is 12.47 against 7.00 for TM2T [arxiv:2306.14795 tab. 4]. The abstract calls the results state of the art, and the text of §4.2 says "competitive" on most metrics [arxiv:2306.14795 §1] [arxiv:2306.14795 §4.2]. ### Coordinates as plain text Used by [Shikra](/papers/2306.15195). Shikra keeps the usual bridge: a frozen CLIP ViT-L/14, one trainable linear layer giving 256 visual tokens, and Vicuna at 7B or 13B [arxiv:2306.15195 §5.1]. It adds no coordinate vocabulary. A box is written as `[x_min,y_min,x_max,y_max]` and a point as `[x_center,y_center]`, normalised to the image size and kept to three decimals, and it is tokenised like any other text [arxiv:2306.15195 §5.2]. Inputs can contain coordinates as well as outputs. In a toy comparison trained only on referring-expression data, models that write numbers beat models with extra coordinate tokens on every split [arxiv:2306.15195 tab. 2]. The cost is length: numbers take more tokens per box, which matters in dense scenes [arxiv:2306.15195 §6.2]. Shikra-7B reaches 87.01 on RefCOCO val, a generalist result, against 90.56 for the specialist G-DINO-L [arxiv:2306.15195 tab. 3]. The official code is CC BY-NC 4.0, so an implementation should not copy from it. ### Remote-sensing chat with boxes written as text Used by [GeoChat](/papers/2311.15826). The design follows LLaVA-1.5. A CLIP ViT-L/14 has its position embeddings interpolated to take 504 x 504 images, which gives 1296 patch tokens. A two-layer MLP maps them into a Vicuna-v1.5 7B model that is tuned with LoRA, while the encoder and the MLP stay frozen [arxiv:2311.15826 §3.1] [arxiv:2311.15826 §3.2]. Three additions set it apart. Boxes are plain text: four coordinates scaled to [0, 100] and a rotation angle, and region boxes can appear in the prompt as well as in the answer [arxiv:2311.15826 §3.1]. Three task tokens select the answer style: grounding, describing the object in a given box, and locating a named phrase [arxiv:2311.15826 tab. 1]. The training pairs are written by a text-only model from detection labels over aerial images, about 306k in the training split [arxiv:2311.15826 §4] [arxiv:2311.15826 tab. 1]. On zero-shot scene classification, the authors report 84.43% on UCMerced and 72.03% on AID, against 68.00% and 51.00% for LLaVA-1.5 [arxiv:2311.15826 tab. 5]. Their grounding benchmark is built with the same pipeline as the training data, so it is not an independent test [arxiv:2311.15826 §5.4]. The paper's LoRA targets differ from the released code (see the [card's Paper vs code](/papers/2311.15826)). ### Prompt-driven encoder-decoder Used by [Florence-2](/papers/2311.06242). One sequence-to-sequence model takes an image and a text prompt and generates the response as text. The tokenizer vocabulary is extended with 1,000 quantized coordinate bins. Boxes are written as two corners, text regions as four corners and referring-segmentation masks as polygons, all as sequences of these bins [arxiv:2311.06242 §3]. A DaViT backbone turns the image into visual tokens. A linear projection and LayerNorm map them to the decoder width, and they are placed before the prompt embeddings to form the encoder input [arxiv:2311.06242 §3]. Every task uses the same cross-entropy loss over the target tokens [arxiv:2311.06242 eq. 1] [arxiv:2311.06242 §3]. The paper reports two sizes: base at 0.23B and large at 0.77B parameters [arxiv:2311.06242 tab. 4]. Zero-shot, the large model reaches 135.6 CIDEr on COCO captions and 84.4 R@1 on Flickr30k grounding [arxiv:2311.06242 tab. 4]. The baselines, Flamingo 80B and Kosmos-2 1.6B, are as the authors report them [arxiv:2311.06242 §6.2]. ### Pretrained ViT and multilingual encoder-decoder, scaled together Used by [PaLI](/papers/2209.06794). A plain Vision Transformer's patch features, without pooling, feed an mT5 encoder-decoder, and the prompt selects the task. Every task is image and text in, text out, with no task heads [arxiv:2209.06794 §3.1]. The authors argue that earlier large models scaled the language side far more than the vision side, so PaLI-17B pairs a 4B-parameter ViT with mT5-XXL [arxiv:2209.06794 app. A.1]. Pretraining mixes eight tasks over WebLI, a corpus of about 10B images and 12B alt-texts in 109 languages, of which the top 10% by cross-modal similarity are kept [arxiv:2209.06794 §3.2] [app. B]. The corpus is not released, and the largest run used 1,024 TPUv4 chips for 7 days [arxiv:2209.06794 app. A.1]. Results include 149.1 CIDEr on COCO Karpathy test [arxiv:2209.06794 tab. 1], 84.3 on VQAv2 test-dev [arxiv:2209.06794 tab. 3], and 53.6 average CIDEr over 35 languages on Crossmodal-3600, against 47.0 for the 3B model [arxiv:2209.06794 tab. 2]. The paper's text says the visual tokens go to the encoder and also reach the decoder through cross-attention, without saying how they combine with the text [arxiv:2209.06794 §3.1]. Treat that interface as open. Since the largest run is so costly, reproduce a smaller variant first. ### Small language model with a vision adapter Used by [Phi-3.5-Vision](/papers/2404.14219). A CLIP ViT-L/14 encoder feeds phi-3.5-mini (4.2B in total), with visual and text tokens interleaved and high-resolution images split into a grid of blocks [arxiv:2404.14219 §7.1]. Pretraining uses 0.5T tokens of interleaved image-text documents, pairs, OCR PDF data and charts, with the loss on text tokens only [arxiv:2404.14219 §7.1]. The authors report 91.3 on ScienceQA test and 43.0 on MMMU validation [arxiv:2404.14219 tab. 5]. The report leaves out learning rates, batch sizes and data mixture proportions [arxiv:2404.14219 §2], so the training recipe cannot be rebuilt from it. The vision evaluations also use settings that differ from published pipelines [arxiv:2404.14219 §7.2.2], and the authors say the model sometimes fails to refuse harmful requests that depend on OCR, such as scam images [arxiv:2404.14219 §7.4]. ### Language model with a pixel decoder for masks Used by [PixelLM](/papers/2312.02228). A LLaVA-based model, LoRA-tuned, writes an answer that contains a group of learned codebook tokens for each target. The hidden states of those tokens, together with CLIP features from two layers, go into a small pixel decoder that produces one mask per target, so no separate SAM model is needed [arxiv:2312.02228 §3.1]. The method is trained on MUSE, a multi-target dataset whose answers are generated with GPT-4V [arxiv:2312.02228 §4.2]. On the paper's MUSE benchmark, the test-overall gIoU and cIoU are 39.2 and 46.3 for PixelLM-7B, against 12.8 and 27.1 for LISA-7B. The paper's TFLOPs estimate is 3.57 against 7.16 [arxiv:2312.02228 tab. 1]. On refCOCO val, LISA scores higher than PixelLM [arxiv:2312.02228 tab. 3]. These are the authors' numbers, and the released code does not train the target refinement loss the paper describes (see the [card's Paper vs code](/papers/2312.02228)). ### Pixel-grounded chat with a mask per named object Used by [GLaMM](/papers/2311.03356). A global CLIP encoder gives scene features that are projected into a Vicuna-7B model with LoRA. A region encoder pools each user box into one vector, which takes the place of a box token in the prompt [arxiv:2311.03356 §3.1]. The LLM writes a `` token wherever it names an object. The hidden state at each `` goes through a projection into a SAM-style pixel decoder, which returns one mask per token, with SAM's image encoder frozen [arxiv:2311.03356 §3.1] [arxiv:2311.03356 app. A.2]. The answer format wraps each grounded phrase in `

` tags followed by `` [arxiv:2311.03356 §3.2]. Training pretrains on GranD for four tasks at once and then finetunes per task [arxiv:2311.03356 app. A.2] [arxiv:2311.03356 app. A.3]. On the GCG benchmark, the test AP50 is 29.2 and the test recall 40.8, against 24.8 and 35.5 for LISA, which the authors adapted to the task [arxiv:2311.03356 tab. 3]. On referring-expression segmentation, GLaMM scores 79.5 on refCOCO val against 74.9 for LISA-7B, but the table does not name its metric [arxiv:2311.03356 tab. 4]. The paper names CLIP ViT-H/14 for the global encoder, while the released code defaults to a smaller model, so the released weights do not match the stated encoder (see the [card's Paper vs code](/papers/2311.03356)). GranD_f repurposes public referring datasets with GPT-4-written captions [arxiv:2311.03356 §4.5], and the card did not clear those terms for commercial use (see the [card's Pitfalls](/papers/2311.03356)). ### Region vectors in the prompt, with an optional depth connector Used by [SpatialRGPT](/papers/2406.01584). The model is a LLaVA-style VLM on LLaMA2-7B. The image is encoded once, two deconvolution layers upsample the feature map, and the features inside each region mask are averaged into one vector per region [arxiv:2406.01584 app. G.2]. A linear connector maps each region vector into the language space, and it is placed where a region placeholder appears in the text [arxiv:2406.01584 §3.3]. Training samples either the box or the mask of a region at random, and inference uses the mask when one is given, or converts the box to a mask with SAM [arxiv:2406.01584 §3.4]. Relative depth from an off-the-shelf model goes through the same visual encoder, then a separate depth connector, and is averaged per region in the same way. Depth is an extra input: the model also runs without it [arxiv:2406.01584 §3.3]. The training data is synthetic. Objects in about 1M OpenImages photos are detected and given metric depth and camera calibration, placed as boxes in a 3D scene graph, and then turned into questions about size, left and right, above and below, and distance [arxiv:2406.01584 §3.1] [arxiv:2406.01584 §3.2]. The authors report 5M regions and 8M template questions [arxiv:2406.01584 §3.2]. The paper's stage 2 freezes the RGB connector and the LLM, and trains only the depth connector. The released stage-2 script trains the LLM and projector too (see the [card's Paper vs code](/papers/2406.01584)). On the authors' SpatialRGPT-Bench, the 7B model averages 91.78 in qualitative accuracy, against 58.14 for GPT-4V [arxiv:2406.01584 tab. 1]. On BLINK relative depth it reaches 82.3, against 66.9 for GPT-4V-Turbo [arxiv:2406.01584 tab. 4]. GPT-4 judges the answers, so the scores depend on that judge [arxiv:2406.01584 app. J]. ### Visual prompts drawn onto the image Used by [ViP-LLaVA](/papers/2312.00784). Boxes, ellipses, points, triangles, masks, mask contours, arrows and scribbles are alpha-blended into the RGB image, and only the pixels under a prompt change [arxiv:2312.00784 §3.1] [arxiv:2312.00784 eq. 1]. The composite goes through a frozen CLIP ViT-L/14 at 336 px. The paper keeps one early layer and four deeper layers, concatenates them, and maps them with a LayerNorm and a two-layer MLP [arxiv:2312.00784 §3.1] [arxiv:2312.00784 §5.1]. No region encoder is added. In the authors' ablation, the multi-layer features raise Visual7W from 82.44 to 86.04 and PointQA from 70.23 to 70.86 for the 7B base [arxiv:2312.00784 §5.4]. Training uses 520k region-prompt pairs built on existing region-annotated datasets [arxiv:2312.00784 §3.2]. The paper's alpha range and layer indices differ from the released code, which matters if you want the released weights to behave the same (see the [card's Paper vs code](/papers/2312.00784)). ### Numbered arrows drawn on the image, chosen zero-shot Used by [VLMnav](/papers/2411.05755). No model is trained. A depth-based test finds directions the robot can drive in without a collision, and each one becomes a candidate action [arxiv:2411.05755 §3.1]. The candidates are drawn on the camera image as numbered arrows, with a turn-around option labelled 0 [arxiv:2411.05755 §3.3]. One call asks the VLM for a single number, and a second call on the un-annotated image decides when to stop [arxiv:2411.05755 §3.4] [arxiv:2411.05755 §3.5]. On Habitat ObjectNav, the authors report 50.4% success and 0.210 SPL with Gemini Flash, against 24.6% and 0.106 for PIVOT, which samples its actions from a Gaussian [arxiv:2411.05755 tab. 1] [arxiv:2411.05755 §2]. Replacing the depth-based arrows with static, evenly spaced ones gives 33.2% success, and turning off sliding along walls gives 12.9% [arxiv:2411.05755 tab. 1]. The method assumes ground-truth depth and pose from the simulator [arxiv:2411.05755 §4], so the figures do not show how it would fare on a real robot. ### Tool-calling planner Used by [LLaVA-Plus](/papers/2311.05437). Each assistant turn has three fields: `thoughts`, a JSON list of actions with `API_name` and `API_params`, and the reply `value`. An empty action list means no tool is called [arxiv:2311.05437 §2.1]. A skill-oriented sample has four turns: the question, the tool call, the tool output written as text, and the answer [arxiv:2311.05437 eq. 2] [arxiv:2311.05437 §2.2]. The skills cover detection, segmentation, tagging, captioning, OCR, retrieval, generation and editing [arxiv:2311.05437 tab. 2]. The 7B model scores 69.5 on LLaVA-Bench In-the-Wild against 57.1 for LLaVA [arxiv:2311.05437 tab. 4]. The best-scoring setting, "All Tools", writes tool outputs into the context before answering, so it does not test tool choice. The choice evidence is the "Fly" setting, which scores 59.1 with thoughts and 48.7 without them [arxiv:2311.05437 tab. 4]. ### Embedder built from a multimodal LLM Used by [UniME](/papers/2504.17432). The embedding is the normalised hidden state of the last token [arxiv:2504.17432 §3.2]. Stage 1 trains only the LLM, with LoRA and text alone, so that its batch similarities copy those of a text teacher, NV-Embed V2, whose embeddings are computed offline [arxiv:2504.17432 §3.2] [arxiv:2504.17432 §3.4]. Stage 2 trains on image-text pairs with InfoNCE over the positive and eight hard negatives, after removing candidates that score above the positive plus a margin of 0.1 [arxiv:2504.17432 §3.3] [arxiv:2504.17432 §4.1]. On MMEB, the stage-1 model alone reaches 40.3 and stage 2 alone 63.8 with Phi-3.5-V, and both together reach 64.2 [arxiv:2504.17432 tab. 3]. The authors report 70.7 overall with LLaVA-OneVision-7B and 66.6 with LLaVA-1.6, against 63.3 for VLM2Vec on the same base [arxiv:2504.17432 tab. 1]. The baselines were re-run by the authors, and the stage-1 code is not released (see the [card's Paper vs code](/papers/2504.17432)). Think-Then-Embed adds a reasoning step in front of the embedder [arxiv:2510.05014 §4]. A multimodal LLM writes a short reasoning text about the query or the target, and the embedder encodes the original input together with that text. There are three variants: a large teacher reasoner (TTE_t), a small reasoner fine-tuned on the teacher's traces (TTE_s), and one model whose embedding head reads the same pass that writes the reasoning (TTE_u) [arxiv:2510.05014 §4.2] [arxiv:2510.05014 §4.3] [arxiv:2510.05014 §4.4]. Fine-tuning the reasoner matters in the paper's ablation: at 7B the MMEB-V1 score is 73.3 with the fine-tuned reasoner and 68.3 when the backbone writes its own reasoning [arxiv:2510.05014 tab. 3]. On MMEB-V2, the teacher variant reaches 71.5 overall, against 61.2 for the authors' own reproduction of VLM2Vec-V2-7B [arxiv:2510.05014 tab. 2] [arxiv:2510.05014 app. D]. A query is a new item at search time, so the reasoner is in the online path [arxiv:2510.05014 §4.2], and the paper gives no latency. ### Action outputs for robots and vehicles Two papers fine-tune a multimodal LLM to output a few discrete actions, and a conventional controller carries them out. [ManipLLM](/papers/2312.16217) takes an image and a prompt and outputs a contact pixel and two gripper direction vectors. Each direction component is one of 100 bins over [-50, 50], so the output is a classification [arxiv:2312.16217 §3.1.2]. On 10 unseen simulated categories it reaches 0.54 initial-movement success against 0.41 for the best baseline [arxiv:2312.16217 tab. 1]. The released code does not include the impedance search or the test-time adaptation the paper describes (see the [card's Paper vs code](/papers/2312.16217)). [DriveMLM](/papers/2312.09245) outputs one speed decision from four options and one path decision from five, plus an explanation, and a motion planner executes the decision [arxiv:2312.09245 §3.2]. On CARLA Town05 Long, the authors report a driving score of 76.1 against 71.4 for Apollo [arxiv:2312.09245 tab. 4]. The training data is available only on request and no code is released (see the [card's Pitfalls](/papers/2312.09245)). ### Slices and tiles for volumes and whole slides Used by [MedGemma 1.5](/papers/2604.05081). A 4B model with a frozen MedSigLIP encoder takes a 3D scan as a sequence of axial slices, each rescaled to 896 x 896 pixels. A query holds at most 85 slices, which is 21,760 vision tokens, and longer stacks are sampled at equal spacing along the z-axis [arxiv:2604.05081 §2.3.1]. CT slices are mapped to three colour channels with three Hounsfield windows, while MRI volumes are normalised per volume with no window [arxiv:2604.05081 §2.3.1]. A whole-slide image is cut into non-overlapping 896-pixel patches over the tissue, and at most 126 are kept, chosen at random and put back in their original order [arxiv:2604.05081 §2.3.2]. The authors report 3D CT accuracy of 61.1 and 3D MRI accuracy of 64.7, against 58.2 and 51.3 for MedGemma 1 4B [arxiv:2604.05081 tab. 4]. These are single runs at temperature 0, and the training sets are mostly internal, so only the public evaluation sets can be rerun [arxiv:2604.05081 tab. 1] [tab. 2]. The authors say the model is a research starting point and not for clinical use [arxiv:2604.05081 §4]. ### Interleaved features from two vision encoders Used by [Eyes Wide Shut](/papers/2401.06209), which also builds the MMVP benchmark. The authors search for image pairs that CLIP ViT-L/14 scores as nearly identical (cosine similarity above 0.95) while DINOv2 ViT-L/14 scores them as clearly different (below 0.6) [arxiv:2401.06209 §2.1]. The 150 pairs give 300 questions, and a pair counts as correct only when both of its questions are right, so random guessing scores 25% [arxiv:2401.06209 §2.2] [arxiv:2401.06209 §2.3]. Human participants average 95.7% [arxiv:2401.06209 §2.3]. GPT-4V scores 38.7% in one run [arxiv:2401.06209 app. B.3], and most other models are below 25% [arxiv:2401.06209 §2.3]. The best CLIP model on the paper's CLIP-based variant reaches 37.8 [arxiv:2401.06209 tab. 1]. The fix tested on LLaVA is to pass both encoders' tokens to the LLM. In interleaved MoF, each encoder has its own adapter, and the two token sequences alternate in spatial order [arxiv:2401.06209 §4.3]. With LLaVA-1.5 at 224 px, the interleaved model reaches 28.0 on MMVP with 512 visual tokens, against 24.7 for LLaVA-1.5 at 336 px with 576 tokens [arxiv:2401.06209 tab. 3]. At 224 px, the interleaved model gives the LLM 512 visual positions where plain LLaVA gives 256 [arxiv:2401.06209 tab. 3]. On wider benchmarks the 224 px interleaved row is lower on some tasks, for example MMBench at 61.6 against 67.7 [arxiv:2401.06209 tab. 7]. The released scripts run the interleaved path only through the LLaVA-1 setup, and every shipped script uses 336 px, so the 224 px rows are not reproduced by them (see the [card's Paper vs code](/papers/2401.06209)). ### Supervisor VLM that checks gripper changes of a VLA A vision-language-action policy normally runs its predictions without a check. [FPC-VLA](/papers/2509.04018) adds a Qwen2.5-VL 7B supervisor that is queried only when the gripper state changes by more than 0.5 [arxiv:2509.04018 §3.5] [arxiv:2509.04018 §4.1]. It answers Yes or No, and on No it gives a directional correction for x, y, z and yaw [arxiv:2509.04018 §3.5]. Before the supervisor sees an action, the policy's recent predictions are averaged by a dual-stream fusion [arxiv:2509.04018 §3.3]. On SIMPLER WidowX the authors report 64.6 average task success, against 51.8 for CogACT, the best baseline in their table [arxiv:2509.04018 tab. 3]. In their four-task ablation, removing the supervisor lowers task success from 82.1 to 74.4 [arxiv:2509.04018 tab. 7]. The supervisor is trained mostly on synthetic pairs, 100k per robot against 10k annotated entries [arxiv:2509.04018 §4.1]. The card records no official code (see the [card's Paper vs code](/papers/2509.04018)). ### Slow VLM planner with a fast diffusion driver [DualVLN](/papers/2512.08186) splits vision-and-language navigation into two systems. System 2 is a Qwen2.5-VL 7B planner that outputs a pixel goal, or a view-adjustment turn when no future waypoint is visible [arxiv:2512.08186 §3.1]. System 1 is a diffusion transformer that outputs 32 waypoints. It is conditioned on four latent tokens from System 2 and on a 32-token summary of two RGB frames [arxiv:2512.08186 §3.2] [arxiv:2512.08186 §3.3]. The two systems run asynchronously, so the driver keeps updating from newer frames between planner calls [arxiv:2512.08186 §3.2]. On R2R-CE val-unseen with RGB only, the authors report 64.3 success rate against 56.9 for StreamVLN [arxiv:2512.08186 tab. 1]. The timing figures are 0.7 s per planner output and 0.03 s per driver call on one RTX 4090, both author measurements [arxiv:2512.08186 §5.2]. The released code differs from the paper in several details, such as a scale factor on the trajectory and the sampler settings, so a reimplementation should follow the [card's Paper vs code](/papers/2512.08186). ### Frozen VLM hint for a small imitation driver Used by [Zero-shot LLM driving](/papers/2411.14256). A ViT-B/16 policy is trained by imitation on one cone in a hallway. It has one steering and throttle head for each of three route classes, LEFT, MIDDLE and RIGHT [arxiv:2411.14256 §2.1] [arxiv:2411.14256 app. B.1]. A frozen multimodal LLM is not fine-tuned. It sees the same camera frame with a five-step chain-of-thought prompt and answers with one of the three words [arxiv:2411.14256 §2.1] [arxiv:2411.14256 tab. 1]. The policy acts on every frame and keeps the last word it received while a new answer is pending [arxiv:2411.14256 §2.2]. The LLM is slow. Its answers take between 5.9 and 7.8 seconds on the authors' hardware [arxiv:2411.14256 tab. 2], so the car would stall without the cache. On a harder zigzag course with cones and bins, the policy alone succeeds in 40% of real runs, and with an LLM hint the success rate is 63% to 83%, depending on the model [arxiv:2411.14256 tab. 3]. Those are 30 runs per configuration on one course [arxiv:2411.14256 §3.3]. The paper does not say which instruction the policy uses before the first answer [arxiv:2411.14256 §2.2]. ## Post-training with verifiable rewards Two papers train a multimodal model with reinforcement learning on rules that can be checked automatically, rather than on copied answers. Both use GRPO on Qwen-based models and both report their own numbers. [Visual-RFT](/papers/2503.01785) fine-tunes Qwen2-VL with rewards for box overlap and confidence, exact class match for classification, and a format check [arxiv:2503.01785 §3.2]. On one-shot fine-grained classification over four datasets, the authors report an average of 80.3 for Visual-RFT, 51.7 for SFT and 56.0 for the 2B base model [arxiv:2503.01785 tab. 2]. On 2-shot COCO detection with the 2B model, the mAP goes from 19.6 to 41.5 [arxiv:2503.01785 tab. 3]. The paper gives no learning rate and no IoU threshold, and the code sets the threshold to 0.5 with a strict comparison (see the [card's Paper vs code](/papers/2503.01785)). [VideoChat-R1](/papers/2504.06958) fine-tunes Qwen2.5-VL-7B on a joint set of temporal grounding, object tracking and grounded QA, 18,031 samples in total [arxiv:2504.06958 §3.3]. Its rewards are an IoU for time ranges and boxes, an answer-accuracy check, and a format check [arxiv:2504.06958 §3.2]. The authors report Charades-STA mIoU rising from 29.0 to 60.8 and GoT-10k overlap from 12.6 to 43.8 over the base model [arxiv:2504.06958 tab. 1]. In their ablation, three epochs of SFT lower ActivityNet mIoU from 20.6 to 17.3 [arxiv:2504.06958 tab. 3]. The released grounding and tracking code does not include the caption-recall reward or the clue-based inference the paper describes (see the [card's Paper vs code](/papers/2504.06958)). ## Continual instruction tuning Training a model on tasks in sequence makes it forget earlier ones. [SAME](/papers/2602.01990) studies this for LLaVA-v1.5-7B. The vision encoder and the projector are frozen, and only LoRA experts in the language model's feed-forward layers, with a router, are trained [arxiv:2602.01990 §3] [arxiv:2602.01990 §5.1]. The method stores no samples from earlier tasks [arxiv:2602.01990 §1]. It has three parts [arxiv:2602.01990 §4]: a router update limited to the input directions that matter for the current task, expert updates scaled by the earlier tasks' input covariance, and a per-task freeze of experts that the current task barely uses [arxiv:2602.01990 §4.1] [arxiv:2602.01990 §4.2] [arxiv:2602.01990 §4.3]. The author-reported averages are 66.82 on CoIN against 50.58 for MoELoRA [arxiv:2602.01990 tab. 2], 67.12 on UCIT against 52.06 for MoE-LoRA [arxiv:2602.01990 tab. 3], and 46.53 on the authors' ten-task TriGap benchmark against 44.45 [arxiv:2602.01990 tab. 1]. On CoIN, the ablation rises from 50.58 to 61.32 with the router part, to 65.89 with the expert part, and to 66.82 with freezing [arxiv:2602.01990 tab. 4]. The paper reports single runs with no error bars [arxiv:2602.01990 §5.1]. The CoIN baselines are taken from their papers, while the TriGap baselines were re-run [arxiv:2602.01990 tab. 2] [arxiv:2602.01990 tab. 1]. The hyperparameters were picked on a 20% sample of UCIT's training data and then reused for every benchmark [arxiv:2602.01990 app. F]. The released code routes once per forward pass rather than per token, and its per-task reset is never called (see the [card's Paper vs code](/papers/2602.01990)). ## Reasoning before the answer, with a picture Two papers test whether a model should write its reasoning before its answer when the question includes a picture. Both use ScienceQA, and they disagree on the order. The ScienceQA paper ([Lu et al.](/papers/2209.09513)) builds 21,208 multiple-choice science questions, many with a lecture and an explanation, and prompts models to write the answer first, then the lecture and explanation [arxiv:2209.09513 §3] [§4.2]. Two-shot GPT-3 in that format scores 75.17 on the test split, 1.20 points above two-shot GPT-3 without the reasoning [arxiv:2209.09513 tab. 3]. On a 1,000-item test, answer-first output scores 73.6 against 60.6 when the lecture comes first [arxiv:2209.09513 tab. 7]. With gold explanations in the prompt the score reaches 94%, which shows that explanation quality matters but is not a method [arxiv:2209.09513 tab. 6]. [Multimodal-CoT](/papers/2302.00923) splits the job between two models. Each is a T5 initialised from FLAN-Alpaca. The first writes a rationale from the text and frozen ViT-L/32 features, and the second answers from the text plus that rationale [arxiv:2302.00923 §4.1]. The picture enters through single-head attention and a sigmoid gate [arxiv:2302.00923 §4.2]. In a text-only two-stage pilot, the authors read 56% of 50 sampled errors as rationales that mislead the answer step [arxiv:2302.00923 §3.2], and they say the picture features correct 60.7% of those hallucinated-rationale mistakes [arxiv:2302.00923 §3.3]. In a 200M text-only pilot, both rationale-first (69.32) and answer-first (69.68) lower accuracy against no reasoning (81.63) [arxiv:2302.00923 tab. 2]. With the picture features, the Base model (223M) reaches 85.31 and the Large model (738M) 90.45 on the ScienceQA test split [arxiv:2302.00923 tab. 4]. Generated rationales from InstructBLIP and ChatGPT, used in place of annotated ones, give 87.76 on the Large model against 90.45 [arxiv:2302.00923 tab. 7]. ## Inference cost: dropping image tokens Image tokens are a large share of the input. [FastV](/papers/2403.06764) ranks the image tokens by attention after an early layer, keeps the top share and drops the rest for the later layers. It needs no training [arxiv:2403.06764 §4.1]. The headline setting, K=2 and R=50%, keeps the four-benchmark average of LLaVA-1.5-13B at 73.6 with about 55% of the theoretical FLOPs [arxiv:2403.06764 tab. 1]. The one real timing measurement is on A-OKVQA with one-token answers, on a single A40. The 13B model with FastV takes 0.341 s per example against 0.344 s for the unmodified 7B model [arxiv:2403.06764 tab. 4]. The FLOPs figures come from a formula, not from timing [arxiv:2403.06764 §4.2]. The released code ranks tokens by a different score from the one the paper states, and its demo uses different settings from the headline (see the [card's Paper vs code](/papers/2403.06764)). Streaming video adds a memory problem. [HERMES](/papers/2601.14724) splits the KV cache of a streaming video model into three roles by depth: shallow layers favour recent tokens, deep layers keep one anchor per frame, and middle layers blend the two. It keeps a fixed token budget per layer and merges the dropped tokens of deep layers into one summary token [arxiv:2601.14724 §2] [arxiv:2601.14724 §3.2]. It needs no training. With Qwen2.5-VL-7B at a 4K budget, the authors report 79.44 on StreamingBench real-time against 73.31 for the base [arxiv:2601.14724 tab. 1]. The offline gains are mixed: the VideoMME average for the same model falls from 64.52 to 60.63 [arxiv:2601.14724 tab. 4]. The latency ratios in the paper come from a comparison chart with no values in the text [arxiv:2601.14724 §4.3]. ## Object hallucination at decoding time Object hallucination means naming objects that are not in the image [arxiv:2311.16922 §1]. [Visual contrastive decoding](/papers/2311.16922) changes only the sampler. At each step it computes the next-token logits twice, once for the image and once for a copy with Gaussian noise added [arxiv:2311.16922 §3.2] [arxiv:2311.16922 §3.3.1]. It then contrasts them as (1+α)·logit(image) − α·logit(noised image), with α=1 [arxiv:2311.16922 eq. 3] [arxiv:2311.16922 §4.1]. Only tokens whose probability under the clean image is at least β=0.1 times the top probability are kept [arxiv:2311.16922 §3.3.2] [arxiv:2311.16922 eq. 4]. The authors report up to +5.8 accuracy and +7.4 F1 on POPE across LLaVA-1.5, InstructBLIP and Qwen-VL [arxiv:2311.16922 §4.2] [arxiv:2311.16922 tab. 1]. For LLaVA-1.5 on MSCOCO random, accuracy goes from 83.29 to 87.73 [arxiv:2311.16922 tab. 1]. On the MME hallucination subset, LLaVA-1.5 totals 565.33 against 604.66 [arxiv:2311.16922 tab. 2]. On LLaVA-Bench, with answers judged by GPT-4V, Qwen-VL's accuracy score goes from 4.76 to 6.69 [arxiv:2311.16922 tab. 3]. The method needs a second forward pass per generated token. The authors call the overhead minimal, but report no timing or memory numbers [arxiv:2311.16922 §1]. The paper's β sweep on MME scores β=0.5 higher on the hallucination subset, 635.00 against 591.67 at the default 0.1, and the paper does not explain the default [arxiv:2311.16922 app. B]. Its noise is described as a multi-step process with a fixed γ, while the released code uses a different schedule, and the paper does not say which one produced its numbers (see the [card's Paper vs code](/papers/2311.16922)). The tests cover three models at 7B scale, plus 13B variants of two of them [arxiv:2311.16922 app. B]. ## Multi-agent workflows [SurgRAW](/papers/2503.10265) puts several prompted models around a vision-language model, with no training. An orchestrator routes each question to a visual or a knowledge path. Visual answers pass through a panel that checks an instrument-action table and rates the answers, with up to three refinement runs [arxiv:2503.10265 §3.2] [arxiv:2503.10265 §3.3]. Knowledge questions receive retrieved medical text [arxiv:2503.10265 §3.3]. The authors report 61.73% overall for SurgRAW-GPT4o against 31.44% for GPT-4o alone, on their own five-task benchmark of robotic surgery frames [arxiv:2503.10265 tab. 3]. The benchmark is built from GPT-4o questions and reviewed by two clinicians, with no error rate reported [arxiv:2503.10265 §3.1.2]. The repository has no evaluation script, so the accuracy numbers cannot be checked from it (see the [card's Paper vs code](/papers/2503.10265)). ## Safety of multimodal models A picture can carry a request that the text alone would not. [MM-SafetyBench](/papers/2311.17600) builds 1,680 harmful questions in 13 policy scenarios, each paired with three generated images: a Stable Diffusion picture of the key phrase, a typography picture of the phrase, and the two stacked [arxiv:2311.17600 tab. 1] [arxiv:2311.17600 §3.2]. A GPT-4 judge labels each answer safe or unsafe [arxiv:2311.17600 §3.3]. For LLaVA-1.5-7B, the average attack success rate rises from 41.01 with text only to 71.52 with typography, and to 72.14 with the stacked image [arxiv:2311.17600 tab. 3]. A fixed safety sentence placed before the question lowers the stacked-image average to 17.08 [arxiv:2311.17600 tab. 4]. The paper's text gives a different figure for the tiny subset, 15.68, which appears in no table [arxiv:2311.17600 §3.4]. Use the table values. The data is released under non-commercial terms, so reuse the method, not the images or questions (see the [card's Pitfalls](/papers/2311.17600)). Small pixel changes are a second risk. [Schlarmann and Hein](/papers/2308.10741) attack OpenFlamingo 9B white-box with APGD, bounding the change to 1/255 or 4/255 in the infinity norm [arxiv:2308.10741 §4] [arxiv:2308.10741 §5]. At 4/255, the targeted attack reaches 100% success for one chosen caption in both zero-shot and 4-shot settings, and 83.80% zero-shot for the second [arxiv:2308.10741 tab. 2]. Untargeted COCO CIDEr falls from 84.01 to 1.69 zero-shot [arxiv:2308.10741 tab. 3]. The study covers one model and no transfer to other systems. [Omni-SafetyBench](/papers/2508.07173) renders 972 harmful seed prompts into 24 text, image, video and audio combinations, giving 23,328 cases [arxiv:2508.07173 §2.2] [arxiv:2508.07173 tab. 2]. Each response is judged for comprehension, harm and refusal, and only responses the model understood enter the rates [arxiv:2508.07173 §2.3.1]. Of eleven models, the authors report that three exceed 0.6 on both the average Safety-score and the cross-modal consistency score [arxiv:2508.07173 §3.2] [arxiv:2508.07173 tab. 8]. The paper's labels come from a majority of five LLM judges, but the public script uses one judge and does not compute the Safety-score or the consistency score (see the [card's Paper vs code](/papers/2508.07173)). A closed omni-model is audited in a different way. The GPT-4o system card [arxiv:2410.21276 §1] covers speech, image and video inputs, and its safety work centres on voice: the voice-output classifier reaches precision 0.96 and recall 1.0 on English [arxiv:2410.21276 tab. 1], and refusals for speaker identification rise from 0.83 to 0.98 in the authors' table [arxiv:2410.21276 tab. 2]. The audio evaluations are text benchmarks converted to speech with a TTS system, so the scores also reflect the TTS system [arxiv:2410.21276 §3.2]. The card gives no architecture, learning rate or weights, so it documents the evaluation design rather than the method [arxiv:2410.21276 §2]. ## Knowledge questions about a pictured entity [InfoSeek](/papers/2302.11713) asks questions whose answers are not visible in the picture, such as when a building was constructed [arxiv:2302.11713 §1]. It has 8.9K human-written questions and about 1.35M generated from Wikidata [arxiv:2302.11713 §3.1] [arxiv:2302.11713 §3.2]. Zero-shot, the pre-trained models score low. After fine-tuning, PaLI-X reaches 22.1 overall on the Wikidata split and 10.8 on the human split, while an entity recognizer followed by a Wikipedia reader reaches 18.2 on the human split [arxiv:2302.11713 tab. 4] [arxiv:2302.11713 tab. 5]. The metric in the paper differs from the released evaluation code, so the two may not reproduce each other (see the [card's Paper vs code](/papers/2302.11713)). ## Reading text in pictures Reading the words in a photo or a document is a skill separate from describing a scene. [OCRBench](/papers/2305.07895) tests it with 1000 question-answer pairs in five jobs: text recognition (300 items), scene-text VQA (200), document and chart VQA (200), key-information extraction (200) and handwritten maths (100) [arxiv:2305.07895 §3.3] [arxiv:2305.07895 tab. 5]. A reply scores 1 if a ground-truth string appears in it, and 0 otherwise. The final score is the sum over the 1000 items, so its maximum is 1000 [arxiv:2305.07895 §3.1] [arxiv:2305.07895 tab. 3]. The released scorer also ignores letter case, which the paper does not state (see the [card's Paper vs code](/papers/2305.07895)). The authors' zero-shot runs give Gemini 659 and GPT4V 645 out of 1000, with the open Monkey model at 514 [arxiv:2305.07895 tab. 3]. GPT4V refused 84 images on safety grounds, and a refusal scores 0 [arxiv:2305.07895 §3.3]. Substring matching lets a long reply pass if it contains the answer, and it does not normalise dates, so "2 February 2018" fails for "02/02/2018" [arxiv:2305.07895 §3.3] (see the [card's Pitfalls](/papers/2305.07895)). ## Long-video evaluation Video models often pass tests that a few seconds of footage can answer. [EgoSchema](/papers/2308.09126) is a five-option, multiple-choice benchmark of about 5,000 questions on three-minute first-person clips [arxiv:2308.09126 §3.1] [arxiv:2308.09126 §4.2]. The authors measure how much of a clip a human must watch to verify an answer, with a median of about 100 seconds [arxiv:2308.09126 §4.1]. Published video-language models score between 19.6% and 32.1% against 20% chance, while people score about 75% to 76% [arxiv:2308.09126 §4.2]. The answer key is mostly withheld, and the video comes from Ego4D (see the [card's Pitfalls](/papers/2308.09126)). Video-ChatGPT uses a different measure, a GPT-3.5 judge on a five-aspect video chat benchmark. It gives Video-ChatGPT a correctness score of 2.40 on a 1-to-5 scale, the top score among the compared models [arxiv:2306.05424 tab. 1]. The judge is a closed model, so the scores move with the judge [arxiv:2306.05424 tab. 4]. Scores from the two benchmarks are not comparable. [MVBench](/papers/2311.17005) takes a different route: it turns 9 families of static image questions into 20 temporal tasks, each with 200 multiple-choice items drawn from 11 public video datasets [arxiv:2311.17005 §3.1] [§3.2]. The answer is scored as the option letter that follows a fixed prefix, so the scoring does not need a judge model [arxiv:2311.17005 §3.3]. The authors report 51.1 average accuracy for VideoChat2, a Vicuna-7B model trained in three stages, against 35.5 for VideoChat [arxiv:2311.17005 tab. 2]. Frame count and resolution differ across the rows of that table, and GPT-4V was run at 512 x 512 with 16 frames, so the GPT-4V comparison is not like for like [arxiv:2311.17005 tab. 2]. A variant fed a blank video and no LoRA weights scores 34.7, close to VideoChat's 35.5 [arxiv:2311.17005 §5.1]. ## Spatial memory and layout from video Video models are usually tested on what happens in a clip. [VSI-Bench](/papers/2412.14171) tests whether they can recall the layout of a room they have walked through. It has over 5,000 question-answer pairs from 288 indoor videos, taken from the validation sets of ScanNet, ScanNet++ and ARKitScenes [arxiv:2412.14171 §3.1] [arxiv:2412.14171 app. B.1]. The eight tasks cover object count, absolute distance, object size, room size, relative distance, relative direction, route plan and appearance order [arxiv:2412.14171 §3.2]. Multiple-choice answers are scored by accuracy, and numerical answers by mean relative accuracy over ten thresholds [arxiv:2412.14171 §4.1] [arxiv:2412.14171 eq. 1]. The authors report that humans reach 79.2 on a 400-question subset, and the best model, Gemini-1.5 Pro, 48.8 on the same subset [arxiv:2412.14171 tab. 1]. Seven of the twelve open-source models score below the frequency baseline on average [arxiv:2412.14171 §4.2] [arxiv:2412.14171 tab. 1]. The models see different input: Gemini gets the whole video, and open-source models get 8 to 32 uniformly sampled frames [arxiv:2412.14171 app. C.3]. Humans score 94 to 100 on the configurational and appearance-order tasks, and the gap to the best model is narrower on the three measurement tasks [arxiv:2412.14171 §4.2]. Written reasoning does not help on this benchmark. Zero-shot chain-of-thought, self-consistency and tree-of-thoughts all lower the average against no prompting. The authors give about 4% for the first and third and 1.1% for self-consistency, and those drops appear only in a figure [arxiv:2412.14171 §5.2] [arxiv:2412.14171 fig. 8]. Self-explanations of 163 wrong answers put about seven in ten errors under spatial reasoning [arxiv:2412.14171 §5.1]. An explicit map helps on distance questions. Gemini-1.5 Pro is first asked to place each object's centre on a 10 x 10 grid, and then to answer the relative-distance question from that map. Accuracy rises from 46.0 to 56.0 [arxiv:2412.14171 tab. 3]. A 20 x 20 grid gives 54.0 with the model's own map. Maps built from ground-truth annotations give 66.0, or 78.0 at 20 x 20, which the authors treat as an upper bound and not a method [arxiv:2412.14171 tab. 3]. The second-stage prompt is not stated [arxiv:2412.14171 §6.2]. The model places adjacent objects correctly within one grid unit in 64% of cases, and accuracy falls with distance [arxiv:2412.14171 app. B.5]. ## Which input the answer relies on [Sensory PID](/papers/2606.00959) splits what video and audio each contribute to an answer, with the text instruction held fixed, into a video-only part, an audio-only part, a shared part and a synergy part [arxiv:2606.00959 §2.1] [arxiv:2606.00959 eq. 1]. It needs no training. Each stream is replaced with noise in turn, and the answer probabilities are fitted with the BATCH estimator [arxiv:2606.00959 §2.2]. On MUSIC-AVQA, the visual-only term is larger than the audio-only term in most subsets. On the audio-focused subset, the audio term is larger for VITA-1.5 and Qwen2.5-Omni 3B, and nearly equal to the visual term for Qwen2.5-Omni 7B [arxiv:2606.00959 tab. 1]. The study covers three omni-modal models and one benchmark. The same decomposition gives sample weights for vision-language fine-tuning. Over three seeds, Qwen2.5-VL-7B reaches 64.3 on MMStar with the PID weights against 62.0 with uniform weights, a gap of 2.3 points with a 95% interval of [1.4, 3.2] [arxiv:2606.00959 tab. 2] [arxiv:2606.00959 §5.1]. On Gemma3-12B the MMStar score falls from 56.2 to 54.3 [arxiv:2606.00959 tab. 3]. The authors do not claim that the synergy shift causes the gain [arxiv:2606.00959 §5.2]. ## Training data - **Image-text pairs for pretraining.** [LAION-400M](/papers/2111.02114) is about 400 million pairs from Common Crawl alt-text, kept when the CLIP image-text cosine similarity is at least 0.3 [arxiv:2111.02114 §2.1.1]. The paper lists 413M unique samples [arxiv:2111.02114 tab. 1]. It reports no benchmark result for the dataset itself [arxiv:2111.02114 §3]. The dataset's license is not stated, and the CLIP filter is not in the linked code, so the filter cannot be rebuilt from it [arxiv:2111.02114 §2.1.1]. - **Video instruction data.** Video-ChatGPT's 100K pairs are 30% human-written and 70% semi-automatic, the latter made with BLIP-2, GRiT, Tag2Text and GPT-3.5 [arxiv:2306.05424 §4.2] [arxiv:2306.05424 §5.3]. The paper says the data will be released under MIT terms, which the card has not verified [arxiv:2306.05424 §9]. - **Pretraining corpus for a detail-heavy encoder-decoder.** Florence-2's FLD-5B has 126M images and 5.4B annotations [arxiv:2311.06242 §4]. Its labels come from specialist models, including cloud services, in three phases: initial annotation, filtering and iterative refinement [arxiv:2311.06242 §4.2]. The paper does not state the license of FLD-5B itself, and the labelling models' licenses are not stated either [arxiv:2311.06242 §4.1]. - **Medical multi-task benchmark.** MultiMedBench, used by Med-PaLM M, has 14 tasks over 12 datasets and more than one million samples [arxiv:2307.14334 §3]. Report generation on MIMIC-CXR is 59.90% of the fine-tuning mix [arxiv:2307.14334 tab. A.1]. The terms of the clinical datasets were not checked, so check each one before reuse. - **Multi-target segmentation data.** PixelLM's MUSE has 246k question-answer pairs over about 910k LVIS instances [arxiv:2312.02228 §4.1]. Its labels are made with GPT-4V and checked with GPT-4V and humans [arxiv:2312.02228 §4.2]. - **Instruction samples from captions and boxes.** LLaVA's 158K samples are 58K conversations, 23K detailed descriptions and 77K reasoning questions, written by GPT-4 from COCO captions and boxes, without the image [arxiv:2304.08485 §3]. The project page lists the data as CC BY-NC 4.0, so do not use it for commercial training without a separate licence check [arxiv:2304.08485 Pitfalls]. - **Referential dialogue.** Shikra's generated set has 5,922 question-and-answer pairs, written by GPT-4 from Flickr30K captions and boxes rather than from images [arxiv:2306.15195 §5.3.2]. - **Science multiple choice.** ScienceQA has 21,208 questions, split 60:20:20 into train, validation and test [arxiv:2209.09513 §3] [app. A.3]. Its licence is CC BY-NC-SA 4.0, for non-commercial research only, so the dataset cannot be reused commercially [arxiv:2209.09513 app. A.1]. - **Clinical sets.** Most of MedGemma 1.5's training and several evaluation sets are internal, so only the public sets in its table can be reproduced [arxiv:2604.05081 tab. 1] [tab. 2]. - **Speech-text instruction pairs.** SpeechInstruct's cross-modal part turns about 9 million English ASR pairs into recognition and synthesis tasks, with instruction wordings written by GPT-4 [arxiv:2305.11000 §3.1]. The chain-of-modality part converts 37,969 text instruction responses to speech with a generator trained on LibriSpeech [arxiv:2305.11000 §3.2]. The card records no licence for the dataset, so check the source corpora before reuse. - **Synthetic route instructions from video.** [InstruGen](/papers/2411.11394) takes routes through 3,879 YouTube house tours, labels rooms with CLIP and estimates the action between rooms with an inverse model [arxiv:2411.11394 §3.1]. A multimodal model writes the instruction. A consistency check then extracts room-and-action pairs from the text and regenerates the instruction when they do not match the route [arxiv:2411.11394 §3.2]. The paper does not state how pairs are compared, how many retries are allowed, or how many instructions are generated [arxiv:2411.11394 §3.2] [arxiv:2411.11394 §4.1]. - **Question-and-answer pairs for robots in crowds.** [Social-LLaVA](/papers/2501.09024) annotates about 2K robot-human scenes from the SCAND navigation data, with over 40K questions and answers on perception, prediction, reasoning, action and explanation [arxiv:2501.09024 §III-B] [arxiv:2501.09024 §I]. The paper does not say that the set is released, and it does not state the license of SCAND [arxiv:2501.09024 §III-B]. - **Spatial questions generated from 3D scans.** [RoboSpatial](/papers/2411.16537) builds questions from indoor and tabletop scans, with boxes and images from EmbodiedScan [arxiv:2411.16537 §4.1]. Three relation types are generated: free points in a direction from an anchor object, whether a virtual box of the target's size fits, and whether one object lies left, right, above, below, in front of or behind another. Each relation is read in the camera, world or object frame [arxiv:2411.16537 §3.1] [arxiv:2411.16537 §3.2]. Training uses 900,000 relations, and LLaVA-NeXT's average on the authors' validation split rises from 38.1 at 100K relations to 72.4 at the full 3M [arxiv:2411.16537 app. B.1] [arxiv:2411.16537 tab. 9]. The source scan sets carry their own terms, which the card did not check, and the official code is licensed for research and evaluation only (see the [card's Pitfalls](/papers/2411.16537) and the [card's Paper vs code](/papers/2411.16537)). ## When to use which | Situation | Use | Why | |---|---|---| | Training from open components on a limited budget | Frozen LLM with a bridge ([InstructBLIP](/papers/2305.06500)) | Only the Q-Former is trained, with 188M trainable parameters in finetuning, against 1.2B for BLIP-2 finetuning [arxiv:2305.06500 §3.5]. The instruction-tuning run used 16 A100 40GB GPUs for up to 1.5 days [arxiv:2305.06500 §2.6]. | | Video input on a small training budget | Frozen LLM with a linear bridge ([Video-ChatGPT](/papers/2306.05424)) | Only the linear projection is trained, in about 3 hours on 8 A100 40GB GPUs [arxiv:2306.05424 §5.1]. It has the top correctness score in the paper's video chat benchmark [arxiv:2306.05424 tab. 1], but the judge is closed and the data is not in the repository. | | Several input types from image-text data only, for a demo | One embedding space ([PandaGPT](/papers/2305.16355)) | Six input types share one encoder, and the model is trained on image-text pairs only [arxiv:2305.16355 §3]. No benchmark is reported [arxiv:2305.16355 §4], and the released data and weights are non-commercial. | | Long documents or long dialogue histories | Natively multimodal decoder ([Gemini 1.0](/papers/2312.11805)) | Gemini states a 32,768-token context [arxiv:2312.11805 §2]. InstructBLIP's paper states no input text limit, and its code truncates at a default of 128 tokens (see the [card's Hyperparameters](/papers/2305.06500)). | | Millions of tokens of mixed text, video and audio, through a hosted API | Long-context multimodal model ([Gemini 1.5](/papers/2403.05530)) | Pro reaches 100% text needle recall to 530k tokens, above 99.7% at 1M and 99.2% at 10M [arxiv:2403.05530 §5.2.1]. Needle recall is a simple test by the authors' own account, and they pair it with next-token loss curves on long inputs [arxiv:2403.05530 §5.2]. On a 710K-token book, full context was preferred in 78% of side-by-side comparisons with 4k-token retrieval, but the rater was Pro itself [arxiv:2403.05530 §5.2.2]. No weights are released, and the paper gives no parameter count or price [arxiv:2403.05530 app.]. | | Audio input | Gemini 1.0 or PandaGPT | Gemini takes 16 kHz USM features and reports audio benchmarks [arxiv:2312.11805 tab. 11]. PandaGPT takes audio through ImageBind and reports no numbers [arxiv:2305.16355 §4]. | | Speech in and speech out from one model | Discrete speech units ([SpeechGPT](/papers/2305.11000)) | The evidence is a few transcription examples and ten spoken dialogues judged by people, with no metric or baseline [arxiv:2305.11000 §5.2] [arxiv:2305.11000 tab. 2]. The released scripts use learning rates and step counts that differ from the paper (see the [card's Paper vs code](/papers/2305.11000)). | | Chart, document or infographic reading | Natively multimodal decoder, with care | Gemini reports ChartQA 80.8% and DocVQA 90.9% for post-trained Ultra, 0-shot, pixel input only [arxiv:2312.11805 tab. 7]. InstructBLIP does not evaluate these benchmarks. The figures are for the post-trained API model. | | One set of weights for captioning, detection, grounding and segmentation, answered by prompt | Prompt-driven encoder-decoder ([Florence-2](/papers/2311.06242)) | Zero-shot, the large model reaches 37.5 mAP on COCO val2017 detection and 135.6 CIDEr on COCO captions [arxiv:2311.06242 tab. 4]. Generalist fine-tuning reaches 143.3 CIDEr [arxiv:2311.06242 tab. 5]. | | Small parameter budget | Prompt-driven encoder-decoder, base size | The base model is 0.23B parameters and reaches 133.0 CIDEr zero-shot on COCO captions [arxiv:2311.06242 tab. 4]. The two models in this section are evaluated on different tasks, so do not compare their numbers directly. | | Same model writes masks as well as text | Pixel decoder ([PixelLM](/papers/2312.02228)) or a unified vocabulary ([Unified-IO](/papers/2206.08916)) | PixelLM reports TFLOPs about half of LISA's and needs no SAM model [arxiv:2312.02228 tab. 1]. Unified-IO covers boxes, depth and images as tokens and reports 64.3 on GRIT test [arxiv:2206.08916 tab. 3]. | | Chest X-ray report drafting or image-and-text clinical tasks | Fully fine-tuned generalist ([Med-PaLM M](/papers/2307.14334)), as a reference only | The 84B model reaches 53.56% Micro-F1-14 on MIMIC-CXR against 44.20% for prior SOTA [arxiv:2307.14334 tab. 2], but its BLEU-1 is 32.31% against 39.48% [arxiv:2307.14334 tab. 2]. Radiologists ranked the reference report best in 37.14% of cases, and the 84B report was preferred over the reference in 40.50% of pairwise comparisons [arxiv:2307.14334 fig. 4]. | | Medical question answering | Fully fine-tuned generalist, as a reference only | The best Med-PaLM M model reaches 69.68% on MedQA against 86.50% for prior SOTA [arxiv:2307.14334 tab. 2]. | | Medical image classification with no labelled data for the task | Fully fine-tuned generalist, with care | The 562B model reaches 87.68% on tuberculosis detection with 0 training samples, against 92.60% for a SOTA ensemble trained on 138 [arxiv:2307.14334 tab. 4]. The prompt includes a text exemplar [arxiv:2307.14334 §5.2]. | | Reusing a published medical model | Check availability first | Med-PaLM M needs PaLM-E checkpoints, and the paper releases neither code nor weights [arxiv:2307.14334 §4.1]. Use the design as a reference and say which parts are reconstructed. | | Improving checkable perception or grounding with RL | Verifiable-reward fine-tuning ([Visual-RFT](/papers/2503.01785), [VideoChat-R1](/papers/2504.06958)) | Both report large gains over their bases on rule-checkable tasks [arxiv:2503.01785 tab. 2] [arxiv:2504.06958 tab. 1]. Both need an automatic reward that matches the output format, and neither report gives the learning rate. | | Faster inference on a model you cannot retrain | Token pruning ([FastV](/papers/2403.06764)) | No training is needed, and the headline setting keeps the four-benchmark average [arxiv:2403.06764 tab. 1]. The savings are theoretical except for one measured setting [arxiv:2403.06764 tab. 4]. | | Robotic or domain video QA with no training budget | Multi-agent workflow ([SurgRAW](/papers/2503.10265)) | It needs no training and reports 61.73% overall with GPT-4o [arxiv:2503.10265 tab. 3]. Each visual question can take dozens of model calls, and the repository does not run as released (see the [card's Pitfalls](/papers/2503.10265)). | | Testing whether a multimodal model refuses harmful requests | [MM-SafetyBench](/papers/2311.17600) method | The typography picture raises the attack success rate sharply for LLaVA-1.5 [arxiv:2311.17600 tab. 3]. Its data is non-commercial, so build your own test set if you ship. | | Judging long-video understanding | [EgoSchema](/papers/2308.09126) | Models score 19.6% to 32.1% against 20% chance, while people score about 75% [arxiv:2308.09126 §4.2]. Access to Ego4D and the mostly private answer key limit how you can use it. | | Questions that need facts beyond the picture | Entity recognizer plus a reader, or a fine-tuned end-to-end model ([InfoSeek](/papers/2302.11713) baselines) | On the human split, the pipeline reaches 18.2 with FiD against 10.8 for PaLI-X [arxiv:2302.11713 tab. 5] [arxiv:2302.11713 tab. 4]. On the Wikidata split the end-to-end model is ahead, 22.1 against 19.3 [arxiv:2302.11713 tab. 4] [arxiv:2302.11713 tab. 5], so the split decides. | | Asking about a region the user points at, without coordinates | Visual prompts ([ViP-LLaVA](/papers/2312.00784)) | Marks drawn on the image need no coordinate text or region module. At 7B, prompts score 86.04 on Visual7W against 86.36 for coordinate text, and 70.86 on PointQA against 61.4 [arxiv:2312.00784 app.]. | | Answers that need detectors, OCR or generators | Tool-calling planner ([LLaVA-Plus](/papers/2311.05437)) | The 7B model scores 69.5 against 57.1 for LLaVA on In-the-Wild [arxiv:2311.05437 tab. 4]. Each tool call adds a second generation pass, and the paper reports no latency [arxiv:2311.05437 §2.1]. | | One embedding model for image, text and mixed retrieval | Embedder from an MLLM ([UniME](/papers/2504.17432)) | 70.7 on MMEB with LLaVA-OneVision-7B [arxiv:2504.17432 tab. 1]. Stage 2 runs about 1,000 steps at batch 1,024 on 8 A100 80GB GPUs, for 26 to 37 hours [arxiv:2504.17432 §4.1] [arxiv:2504.17432 tab. 6]. | | Aerial or satellite images, with questions about objects and their boxes | [GeoChat](/papers/2311.15826) | Zero-shot 84.43% on UCMerced and 72.03% on AID scene classification, against 68.00% and 51.00% for LLaVA-1.5 [arxiv:2311.15826 tab. 5]. RSVQA-LRBEN is in its training mix, so its VQA numbers are not zero-shot [arxiv:2311.15826 tab. 1] [arxiv:2311.15826 tab. 6]. The code's licence is not stated (see the [card's Paper vs code](/papers/2311.15826)). | | Retrieval where the instruction needs reasoning, and a reasoner can run at query time | Think-Then-Embed ([Think Then Embed](/papers/2510.05014)) | 71.5 overall on MMEB-V2 for the 7B teacher variant, against 61.2 for the authors' VLM2Vec-V2-7B reproduction [arxiv:2510.05014 tab. 2]. The 7B fine-tuned reasoner reaches 68.6 [arxiv:2510.05014 tab. 2]. The reasoner runs on each new query [arxiv:2510.05014 §4.2], and no latency is measured. The paper's margin over the closed model is 0.2 points. | | Grasp point or vehicle decision from images, in simulation | Action outputs ([ManipLLM](/papers/2312.16217), [DriveMLM](/papers/2312.09245)) | ManipLLM scores 0.54 on 10 unseen simulated categories [arxiv:2312.16217 tab. 1]. DriveMLM scores 76.1 driving score in CARLA Town05 Long [arxiv:2312.09245 tab. 4]. DriveMLM is tested only in that simulator, and ManipLLM's real-world test covers 7 categories with 5 trials each [arxiv:2312.16217 tab. 3]. Parts of ManipLLM's method are missing from its released code (see the [card's Paper vs code](/papers/2312.16217)). | | A VLA policy that should catch its own gripper mistakes, with a checker that is queried only at gripper changes | VLM supervisor ([FPC-VLA](/papers/2509.04018)) | 82.1 against 74.4 task success with and without the supervisor on four tasks [arxiv:2509.04018 tab. 7]. Training pairs are mostly synthetic, and the timing of 0.176 s per step does not state its hardware [arxiv:2509.04018 §4.1]. | | Real-time navigation where a large planner is too slow to run on every step | Slow VLM planner with a fast diffusion driver ([DualVLN](/papers/2512.08186)) | 64.3 success rate on R2R-CE val-unseen with RGB only, against 56.9 for StreamVLN [arxiv:2512.08186 tab. 1]. The planner runs at 0.7 s and the driver at 0.03 s per call on one RTX 4090 [arxiv:2512.08186 §5.2]. | | Fewer invented objects from an existing vision-language model, no retraining | Contrastive decoding ([Visual contrastive decoding](/papers/2311.16922)) | Up to +5.8 accuracy on POPE across three models [arxiv:2311.16922 §4.2]. It needs a second forward pass per generated token, and the paper reports no timing [arxiv:2311.16922 §1]. | | Long streaming video on a fixed GPU memory budget, no retraining | KV-cache pruning ([HERMES](/papers/2601.14724)) | Qwen2.5-VL-7B at 4K tokens reaches 79.44 on StreamingBench real-time, against 73.31 for the base [arxiv:2601.14724 tab. 1]. Offline VideoMME falls for that model [arxiv:2601.14724 tab. 4]. | | Probing a model's safety across modalities | Parallel benchmark ([Omni-SafetyBench](/papers/2508.07173)) or the perturbation study ([Schlarmann and Hein](/papers/2308.10741)) | Omni-SafetyBench tests 24 variations per request, but only three of eleven models pass 0.6 on both averages [arxiv:2508.07173 tab. 8]. The perturbation study is white-box and bounded at 4/255, and it measures attack success, not real-world risk [arxiv:2308.10741 §4] [arxiv:2308.10741 tab. 2]. | | Checking whether a model answers from the picture or the sound | Sensory PID ([Fang et al.](/papers/2606.00959)) | It needs no training and runs on answer probabilities [arxiv:2606.00959 §2.2]. On the evaluated subsets the visual term is larger, but the study covers three models and one benchmark [arxiv:2606.00959 tab. 1]. | | A simple vision assistant from open parts, one linear layer | LLaVA ([LLaVA](/papers/2304.08485)) | Stage 1 trains only the projection, and stage 2 tunes the projection and the LLM on 158K instruction samples [arxiv:2304.08485 §4.2]. The 85.1 score is a GPT-4 judgement against a reference answer [arxiv:2304.08485 tab. 4]. The instruction data is non-commercial per the project page. | | Cheap alignment on a small curated set | MiniGPT-4 ([MiniGPT-4](/papers/2304.10592)) | Stage 2 takes about seven minutes on one A100 [arxiv:2304.10592 §3.3]. Human judgements cover 100 images, 25 per task [arxiv:2304.10592 §4.2]. Long descriptions hallucinate more: CHAIR_i is 7.2 for short prompts and 9.6 for long ones [arxiv:2304.10592 tab. 5]. | | Captioning with a small trainable part on frozen CLIP and GPT-2 | Frozen CLIP prefix ([ClipCap](/papers/2111.09734)) | The transformer variant has 43M parameters, trains for 6 hours on one GTX 1080, and reaches 113.08 CIDEr on COCO against 124.12 for Oscar, which uses object tags [arxiv:2111.09734 tab. 1 (C)]. | | Pointing at objects by box or point, in text | Coordinates as text ([Shikra](/papers/2306.15195)) | Shikra-7B reaches 87.01 on RefCOCO val, against 90.56 for the specialist G-DINO-L [arxiv:2306.15195 tab. 3]. Numbers beat extra coordinate tokens in the toy comparison, at a cost in tokens [arxiv:2306.15195 tab. 2] [arxiv:2306.15195 §6.2]. The code is CC BY-NC 4.0. | | Many languages and captioning at scale | PaLI ([PaLI](/papers/2209.06794)) | 53.6 average CIDEr over 35 languages for the 17B model, against 47.0 for 3B [arxiv:2209.06794 tab. 2]. Training used 1,024 TPUv4 chips for 7 days, and WebLI is not released [arxiv:2209.06794 app. A.1]. | | One sequence-to-sequence model for captioning, VQA, grounding and images | OFA ([OFA](/papers/2202.03052)) | 82.0 on VQA test-std and 91.2 on SNLI-VE test [arxiv:2202.03052 tab. 2]. Not best on every task: XLNet is ahead on SST-2 in the unimodal table [arxiv:2202.03052 tab. 6]. | | Science questions where the picture matters | Multimodal-CoT ([Multimodal-CoT](/papers/2302.00923)) or the answer-first format ([Lu et al.](/papers/2209.09513)) | Multimodal-CoT Base reaches 85.31 with a 223M model [arxiv:2302.00923 tab. 4]. The two papers disagree on the order of reasoning and answer (see Contradictions). ScienceQA is non-commercial. | | Volumes or whole-slide pathology, research use only | MedGemma 1.5 ([MedGemma 1.5](/papers/2604.05081)) | 3D MRI accuracy 64.7 against 51.3 for MedGemma 1 4B [arxiv:2604.05081 tab. 4]. Single runs, and the authors say the model is not for clinical use [arxiv:2604.05081 §4]. Gemini numbers cannot be reproduced from the paper [arxiv:2604.05081 §3]. | | On-device model that reads images | Phi-3.5-Vision ([Phi-3](/papers/2404.14219)) | 4.2B in total, with 91.3 on ScienceQA test and 43.0 on MMMU validation, as the authors report [arxiv:2404.14219 tab. 5]. The learning rates and batch sizes are not stated [arxiv:2404.14219 §2]. | | Video questions that need several frames | MVBench ([MVBench](/papers/2311.17005)) | 20 temporal tasks of 200 items each, scored without a judge. VideoChat2 reaches 51.1 against 35.5 for VideoChat [arxiv:2311.17005 tab. 2]. | | Tasks arrive one after another and old data cannot be replayed | Stabilised mixture of LoRA experts ([SAME](/papers/2602.01990)) | 66.82 on CoIN against 50.58 for MoELoRA [arxiv:2602.01990 tab. 2]. These are single runs, and the released code departs from the paper's equations (see the [card's Paper vs code](/papers/2602.01990)). | | Measuring whether a model can read text in photos and documents | [OCRBench](/papers/2305.07895) | 1000 items in five jobs, scored by checking for the answer string, with no judge model. The authors report 659 for Gemini and 645 for GPT4V [arxiv:2305.07895 tab. 3]. Token caps differ between the released scripts and the paper does not state them (see the [card's Pitfalls](/papers/2305.07895)). | | Answers that must point at each object they name, with a mask per phrase | Pixel-grounded chat ([GLaMM](/papers/2311.03356)) | 29.2 test AP50 on GCG against 24.8 for LISA, both author-reported [arxiv:2311.03356 tab. 3]. The training data has unresolved licence terms, and the public repository has no pretraining script (see the [card's Pitfalls](/papers/2311.03356)). | | One model that takes interleaved images and text, prompted with a few examples, and can also output pictures | Continuous image embeddings ([Emu2](/papers/2312.13286)) | 68.8 VQAv2 at 16 shots [arxiv:2312.13286 tab. 1] and 84.9 for Emu2-Chat [arxiv:2312.13286 tab. 2], both author-reported. The weights are listed as non-commercial in the code repository's README, although the code itself is Apache-2.0 (see the [card's Pitfalls](/papers/2312.13286)). | | Image input added to an existing 7B LLaMA, with a two-stage recipe | [mPLUG-Owl](/papers/2304.14178) | 66 of 82 OwlEval answers rated A or B against 54 for MiniGPT-4, a manual rating with no variance [arxiv:2304.14178 §4.2]. Stage-1 settings conflict in the paper, and the repository has no stage-1 code [arxiv:2304.14178 app. A]. | | Several images, or worked examples in the prompt, for one model | In-context instruction tuning ([Otter](/papers/2305.03726)) | On MME cognition with two-shot evaluation, 1,430.3 for a model trained with two-shot examples against 1,310.7 without [arxiv:2305.03726 tab. 10]. The data comes from MIMIC-IT, which the authors say includes faces and personal information [arxiv:2305.03726 §4.3]. | | Detection, segmentation or grounding from one prompt-driven decoder with a learned tokenizer | [VisionLLM](/papers/2305.11175) | 44.6 box AP with ResNet-50 and 60.2 with InternImage-H [arxiv:2305.11175 tab. 1]. The linked repository has no implementation of the method at the pinned commit (see the [card's Paper vs code](/papers/2305.11175)). | | Text-to-motion and motion-to-text from one model | [MotionGPT](/papers/2306.14795) | Text-to-motion FID is 0.232 against 0.116 for T2M-GPT, so MotionGPT is not best on every metric [arxiv:2306.14795 tab. 3]. Motion-to-text BLEU@4 is 12.47 against 7.00 for TM2T [arxiv:2306.14795 tab. 4]. | | Visual details CLIP misses, such as direction, counting or colour, when a second vision encoder is acceptable | Interleaved CLIP and DINOv2 tokens ([Eyes Wide Shut](/papers/2401.06209)) | 28.0 on MMVP at 224 px with 512 visual tokens, against 24.7 for LLaVA-1.5 at 336 px [arxiv:2401.06209 tab. 3]. Some wider benchmarks drop, for example MMBench [arxiv:2401.06209 tab. 7]. | | Spatial questions over indoor video, or a test of layout memory | VSI-Bench ([Thinking in Space](/papers/2412.14171)) | Humans reach 79.2 on the 400-question subset, the best model 48.8 [arxiv:2412.14171 tab. 1]. A 10 x 10 map first raises relative-distance accuracy from 46.0 to 56.0 for Gemini-1.5 Pro [arxiv:2412.14171 tab. 3]. Written step-by-step reasoning does not raise the average [arxiv:2412.14171 §5.2]. | | Object search in a building, with no training data, a depth sensor and a hosted VLM | Numbered arrows on the image, chosen zero-shot ([VLMnav](/papers/2411.05755)) | 50.4% success on Habitat ObjectNav with Gemini Flash, against 24.6% for PIVOT [arxiv:2411.05755 tab. 1]. The method assumes simulator depth and pose, and disabling wall sliding drops success to 12.9% [arxiv:2411.05755 §4] [arxiv:2411.05755 tab. 1]. Each step makes two VLM calls [arxiv:2411.05755 §3.5]. | | A small imitation driver with little data, and a slow multimodal model that can advise on the same camera frame | Frozen VLM hint, cached between answers ([Zero-shot LLM driving](/papers/2411.14256)) | 63% to 83% success against 40% for the policy alone, over 30 real runs per setting on one course [arxiv:2411.14256 tab. 3]. Each LLM answer takes 5.9 to 7.8 seconds [arxiv:2411.14256 tab. 2], so the policy must act at sensor rate. | | Route instructions for a vision-language navigation agent, with few hand-labelled routes | Generated instructions with a consistency check ([InstruGen](/papers/2411.11394)) | Success rate on R2R val-unseen rises from 67.42% without the checks to 71.10% with them, though the best row also has manual review [arxiv:2411.11394 tab. 1]. The automatic-only row reaches 70.9%. The linked repository holds only a README (see the [card's Paper vs code](/papers/2411.11394)). | | Reasoning about crowds in language, with a domain question set and human judges | Fine-tuned VLM on a domain question set ([Social-LLaVA](/papers/2501.09024)) | Human scores of 3.95 to 4.19 on five task types, against 2.77 to 3.87 for prompted GPT-4V and Gemini 1.5 Pro [arxiv:2501.09024 tab. 1]. The comparison uses 50 items and fifteen raters, with no variance or test reported [arxiv:2501.09024 §IV-B]. | | Medical images, long patient records or medical questions, with a hosted large model; a reference only | Tuned Gemini for medicine ([Med-Gemini](/papers/2404.18416)) | 91.1 on MedQA against 90.2 for GPT-4 with MedPrompt [arxiv:2404.18416 tab. 1]. The weights are not released, and the multimodal learning rate, step count and batch size are not stated (see the [card's Hyperparameters](/papers/2404.18416)). | | Questions about where a named object is, or how far away it is, with a box or mask given | Region vectors with a depth connector ([SpatialRGPT](/papers/2406.01584)) | 91.78 qualitative accuracy against 58.14 for GPT-4V on the authors' bench, judged by GPT-4 [arxiv:2406.01584 tab. 1]. Direct-distance success is 41.2% for the 7B model [arxiv:2406.01584 tab. 1], so distance stays weak. A box is turned into a mask with SAM at inference [arxiv:2406.01584 §3.4]. The code is Apache-2.0, but its stage 2 differs from the paper (see the [card's Paper vs code](/papers/2406.01584)). | | Spatial relations for a robot's camera, with a generator for training data | Synthetic spatial questions ([RoboSpatial](/papers/2411.16537)) | LLaVA-NeXT rises from 30.3 to 60.5 on the authors' validation split [arxiv:2411.16537 tab. 3], and from 23.7% to 52.6% on a small real-robot test [arxiv:2411.16537 tab. 6]. The official code is for research and evaluation only, and the paper does not specify the direction test [arxiv:2411.16537 §3.2.1], so a commercial build must write its own generator (see the [card's Pitfalls](/papers/2411.16537)). | ## Implementation sketch The shared core is: encode the image or video, map the features into the decoder's input space, prepend them to the text, and run the decoder. The variants differ in the bridge, in which parts train, and in what the decoder writes. The Gemini report does not describe its encoder or bridge, so the sketch follows InstructBLIP. Florence-2 differs in one place: its visual vectors go into the encoder of an encoder-decoder, and a decoder generates the answer [arxiv:2311.06242 §3]. Video-ChatGPT replaces the Q-Former with pooling and one linear layer [arxiv:2306.05424 §3.1]. PandaGPT replaces the encoder with a shared embedding space and adds a single vector per input [arxiv:2305.16355 §3]. Med-PaLM M updates every weight, so there is no frozen part to skip [arxiv:2307.14334 §4.2]. ViP-LLaVA changes the input rather than the bridge: it draws the marks onto the pixels before encoding [arxiv:2312.00784 §3.1]. UniME reads the last token's hidden state as the embedding [arxiv:2504.17432 §3.2]. LLaVA and MiniGPT-4 keep the sketch's linear map and drop the query bridge [arxiv:2304.08485 §4.1] [arxiv:2304.10592 §3]. OFA and PaLI have no separate bridge: OFA writes image codes and box tokens into one vocabulary [arxiv:2202.03052 §3.1], and PaLI feeds ViT patch features to an mT5 encoder-decoder, where the interface is not fully specified [arxiv:2209.06794 §3.1]. ClipCap keeps a bridge but makes it a small mapper that writes prefix vectors from one CLIP embedding [arxiv:2111.09734 §3.1]. SpeechGPT has no bridge at all: speech units are vocabulary tokens [arxiv:2305.11000 §4.2]. SpatialRGPT adds one bridge input per region: the mask-averaged features of that region, placed where the prompt names it, plus an optional depth vector from a second connector [arxiv:2406.01584 §3.3]. ```pseudocode # Shared core: image features enter as input vectors [arxiv:2305.06500 §2.3] # Input: image, text tokens (the instruction or prompt) # Output: logits over the next text token def multimodal_forward(image, text_tokens): # Frozen image encoder, ViT-g/14 in InstructBLIP [arxiv:2305.06500 §2.6] image_feats = image_encoder(image) # (N, d_v) # Bridge: learned queries read the image and the instruction # [arxiv:2305.06500 §2.3]. Trainable in InstructBLIP. Gemini's bridge # is not described [arxiv:2312.11805 §2] query_out = bridge(queries, text_tokens, image_feats) # (K, d_q) # Linear map into the decoder's input space [arxiv:2305.06500 §2.3] visual_vecs = linear_projection(query_out) # (K, d_llm) # Prepend the visual vectors to the embedded text [arxiv:2305.06500 §2.3] embeds = concat(visual_vecs, embed(text_tokens)) # Frozen decoder predicts the next token [arxiv:2305.06500 §2.2] return decoder(embeds) ``` ## Contradictions - **FlanT5-XL size, within InstructBLIP.** The method section gives FlanT5-XL as 3B parameters [arxiv:2305.06500 §2.6]. The zero-shot results section calls the same model 4B [arxiv:2305.06500 §3.1]. The paper does not resolve this. - **TextVQA across papers.** InstructBLIP's FlanT5-XL reaches 46.6 on TextVQA as a held-out, zero-shot dataset [arxiv:2305.06500 tab. 1]. Gemini Ultra reaches 82.3% on TextVQA (val), 0-shot, pixel input only, using the post-trained API model [arxiv:2312.11805 tab. 7]. These are different models, splits and settings, so the numbers are not comparable. Neither report reconciles them. - **Parameter counts, within Florence-2.** The text gives 232M for base and 771M for large [arxiv:2311.06242 §6.1]. The layer breakdown in [app. C] sums to 230M and 770M [arxiv:2311.06242 app. C]. The paper does not explain the gap, so quote the table or the text, and say which. - **Frozen encoder, within Florence-2.** The text says freezing the encoder does not affect image-level tasks [arxiv:2311.06242 §6.5]. The ablation shows the same encoder at 6.9 COCO AP against 19.7 by default, while caption CIDEr rises slightly [arxiv:2311.06242 tab. 12]. The claim holds for captions only. - **Count of LAION-400M, within the dataset paper.** The abstract and introduction say 400 million pairs [arxiv:2111.02114 §1]. Table 1 lists 413M unique samples [arxiv:2111.02114 tab. 1]. The paper does not say which count is released. - **Clinical errors per report, within Med-PaLM M.** The introduction and the radiologist results give 0.25 clinically significant errors per report for the best model [arxiv:2307.14334 §1] [arxiv:2307.14334 §6.3.2]. The appendix table gives 0.23 for significant clinical errors and 0.25 for total clinical errors, both at 84B [arxiv:2307.14334 tab. A.8]. The 0.25 in the text matches the total-clinical-errors row, not the significant-errors row. The appendix text also calls the 12B model the best performer, on total errors, while the table shows the 84B model with the fewest clinical errors [arxiv:2307.14334 app. A.4] [arxiv:2307.14334 tab. A.8]. - **Architecture naming, within Med-PaLM M.** The introduction calls the model sequence-to-sequence [arxiv:2307.14334 §1]. The PaLM base is a decoder-only Transformer [arxiv:2307.14334 §4.1]. The two descriptions cannot both be right, and the paper does not reconcile them. - **Safety-prefix figure, within MM-SafetyBench.** The text gives 77.33 to 15.68 for LLaVA-1.5-7B on the tiny set [arxiv:2311.17600 §4.5]. The table value 77.33 belongs to the Vicuna-7B row of another table, and 15.68 appears in no table [arxiv:2311.17600 tab. 5]. The default-set table gives 72.14 to 17.08 [arxiv:2311.17600 tab. 4]. - **Token-pruning claim, within FastV.** The abstract and the results text say the headline setting costs no performance. The MME total drops from 1143.5 to 1129.8 for InstructBLIP at the same setting, and the paper says it needs K=5 to close the gap [arxiv:2403.06764 tab. 2] [arxiv:2403.06764 §5.3]. - **Token-fusion gain, within PixelLM.** The text says token fusion gives up to 2.4% cIoU on MUSE val [arxiv:2312.02228 §5.5]. The ablation table gives a gain of 2.1 [arxiv:2312.02228 tab. 4]. - **Headline gap, within UniME.** The text gives a 10.3% gain over VLM2Vec with LLaVA-1.6 [arxiv:2504.17432 §4.3]. Table 1 gives 66.6 against 63.3, a gap of 3.3 points [arxiv:2504.17432 tab. 1]. The paper does not reconcile the two. - **Visual7W, within ViP-LLaVA.** The main table gives 87.91 for the 13B model [arxiv:2312.00784 §5.2]. The GPT-4V-data appendix table gives 88.28 for the 13B model [arxiv:2312.00784 app.]. The paper does not explain the difference. - **LLaVA accuracy, within DriveMLM.** The text gives 40.97% [arxiv:2312.09245 §4.3], and the table gives 22.92% for the same model [arxiv:2312.09245 tab. 3]. - **MM-CoT Large on ScienceQA, across papers.** Multimodal-CoT reports 90.45 for its Large model [arxiv:2302.00923 tab. 4]. LLaVA's table lists "MM-CoT Large" at 91.68, marked as a literature value [arxiv:2304.08485 tab. 7]. The two papers do not explain the gap. - **Order of reasoning and answer on ScienceQA.** The ScienceQA paper finds the answer-first format better: 73.6 against 60.6 when the lecture comes first [arxiv:2209.09513 tab. 7]. LLaVA's ablation finds the reverse on ScienceQA: answer-first gives 89.77 against 90.92 for reasons first [arxiv:2304.08485 tab. 8]. The setups differ in model, data and fine-tuning, and neither paper reconciles them. Multimodal-CoT writes the rationale first [arxiv:2302.00923 §4.1]. - **Multimodal-CoT model size, within the paper.** The text gives 200M and 700M parameters, and Table 4 gives 223M and 738M [arxiv:2302.00923 §5.2] [tab. 4]. The paper does not explain the difference. - **Stage 2 length, within SpeechGPT.** The text gives 2,100 steps for cross-modal instruction tuning [arxiv:2305.11000 §5.1], and the appendix table gives 4,000 [arxiv:2305.11000 app. D]. The paper does not reconcile them. - **FloodNetVQA on TriGap, within SAME.** The text describes the gain on FloodNetVQA as clear [arxiv:2602.01990 §5.2], but the table gives MoE-LoRA 90.41 against 81.09 for SAME there [arxiv:2602.01990 tab. 1]. The paper does not reconcile the two. - **Gemini on document reading, across papers.** The Gemini report gives 90.9% on DocVQA for the post-trained Ultra model [arxiv:2312.11805 tab. 7]. OCRBench scores its Gemini entry, which is gemini-pro-vision, at 128 of 200 on its document VQA items [arxiv:2305.07895 tab. 3] [arxiv:2305.07895 tab. 6]. These are different models, data and scoring rules, so the numbers are not comparable. Neither paper reconciles them. - **Stage-2 trainable parts, within mPLUG-Owl.** The abstract and §1 describe LoRA on the LLM together with the abstractor [arxiv:2304.14178 §1]. §3.2 freezes the abstractor in stage 2 and trains LoRA on the LLM alone [arxiv:2304.14178 §3.2]. - **Stage-1 hyperparameters, within mPLUG-Owl.** The text gives learning rate 0.0001, 2,000 warmup steps, weight decay 0.1 and a batch of 2.1M tokens [arxiv:2304.14178 §4.1]. The appendix table gives 2e-4, 375 steps, 0.01 and 4,096 image-caption pairs [arxiv:2304.14178 app. A]. - **Trainable size, within Otter.** The text gives 1.4B trainable parameters [arxiv:2305.03726 §3.1]. Table 2 gives 1.3B for Otter-Image [arxiv:2305.03726 tab. 2]. - **Video frame count, within Otter.** The text samples 64 frames for training [arxiv:2305.03726 §6.1]. The hyperparameter table gives 8 [arxiv:2305.03726 tab. 11]. - **Location-bin ranges, within VisionLLM.** The main text gives 1,025 location tokens covering -512 to 512 [arxiv:2305.11175 §3.4]. The ablation text describes 257 bins as -128 to 128 and 2,049 bins as -1,024 to 1,024 [arxiv:2305.11175 §4.4]. - **Hardware, within MotionGPT.** The experiments section says 8 V100 GPUs [arxiv:2306.14795 §4.1]. The appendix says 64 V100 GPUs [arxiv:2306.14795 app. H]. - **Headline claim against the table, within MotionGPT.** The abstract calls the results state of the art [arxiv:2306.14795 §1]. The text-to-motion table gives T2M-GPT an FID of 0.116 against 0.232 for MotionGPT [arxiv:2306.14795 tab. 3], and §4.2 says "competitive". - **Video-first input, within VSI-Bench.** The text says video-first input is 2.5% worse [arxiv:2412.14171 app. D]. The table puts video-first at 48.8 and question-first at 46.3 [arxiv:2412.14171 app. D], which reverses the direction. - **Chain-of-thought gain on VideoMME, within VSI-Bench.** The text gives a 1.6% improvement [arxiv:2412.14171 §5.2]. The table gives 77.2 without and 79.8 with chain-of-thought, a 2.6-point gain [arxiv:2412.14171 tab. 2]. - **Self-consistency temperature, within VSI-Bench.** The main text gives 1.0 [arxiv:2412.14171 §5.2]. The appendix gives 0.7 [arxiv:2412.14171 app. B.3]. - **LRBEN zero-shot, within GeoChat.** Table 6 calls GeoChat "not specifically finetuned on target dataset" [arxiv:2311.15826 tab. 6]. Table 1 lists 56k RSVQA-LRBEN training pairs [arxiv:2311.15826 tab. 1]. The paper does not reconcile the two. - **Best open model, within Think-Then-Embed.** The text calls the fine-tuned reasoner the best open model at 2B and 7B [arxiv:2510.05014 §6.1]. Table 2 lists RzenEmbed-v1-7B at 68.9 against 68.6 for TTE_s-7B [arxiv:2510.05014 tab. 2]. The paper does not define "open-source". - **Role of the LLM in the driving hint, within Zero-shot LLM driving.** The main text says the language model gives only a high-level instruction and the policy outputs the steering and throttle [arxiv:2411.14256 §1] [arxiv:2411.14256 §2.1] [arxiv:2411.14256 §4]. The appendix says that, with images and instructions, the LLM provides the steering and throttle values and the probability head is dropped [arxiv:2411.14256 app. B.4]. The paper does not say which setup produced the test results. - **Headline MedQA gain, within Med-Gemini.** The abstract and introduction give 4.6 points over Med-PaLM 2 [arxiv:2404.18416 §1]. The results section gives 4.5 points [arxiv:2404.18416 §4.1]. The paper does not reconcile them. - **Surgical view accuracy, within Med-Gemini.** The main table gives 55.2 for the model [arxiv:2404.18416 tab. 3]. The appendix gives 50.0 [arxiv:2404.18416 tab. A1], and the text's 21% figure matches the appendix value. - **Classification claim, within SpatialRGPT.** The text says the model beats the region-aware baselines on classification [arxiv:2406.01584 §4.2]. In the classification table, RegionGPT-7B is higher than SpatialRGPT-7B on both mAP and accuracy, and only the VILA-based variants are higher [arxiv:2406.01584 tab. 3]. - **BLINK gain, within SpatialRGPT.** The text says the gain over GPT-4V-Turbo on BLINK is more than 20% [arxiv:2406.01584 §4.2]. The 8B model gives 87.9 against 66.9, which fits that claim. The 7B model gives 82.3, a 15.4-point gap [arxiv:2406.01584 tab. 4]. - **Commonsense claim, within RoboSpatial.** The text says trained models keep or slightly improve performance on every general benchmark [arxiv:2411.16537 app. D.3]. The MME cognition score falls from 305.4 to 293.2 [arxiv:2411.16537 tab. 10]. ## Papers - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](/papers/2111.02114): an open set of about 400M CLIP-filtered image-text pairs with embeddings and search indexes, without a model benchmark. - [Unified-IO: A unified model for vision, language, and multi-modal tasks](/papers/2206.08916): one T5-style encoder-decoder that writes images, boxes, masks and depth as discrete tokens, trained on 95 datasets. - [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](/papers/2305.06500): instruction-tunes a BLIP-2 model with a frozen image encoder and LLM, training only an instruction-aware Q-Former. - [PandaGPT: One Model To Instruction-Follow Them All](/papers/2305.16355): joins a frozen ImageBind encoder to a frozen Vicuna-13B through one linear projection and LoRA, for six input types, with example outputs only. - [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](/papers/2306.05424): a frozen CLIP encoder and frozen Vicuna-7B joined by one trainable linear layer over pooled frame features, tuned on 100K video-instruction pairs. - [Towards Generalist Biomedical AI](/papers/2307.14334): Med-PaLM M, a PaLM-E model fully fine-tuned on MultiMedBench with text prompts, with a radiology report and classification evaluation. - [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](/papers/2308.09126): a five-option benchmark on three-minute Ego4D clips, with a temporal-certificate measure of how much video each question needs. - [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](/papers/2311.06242): a prompt-driven encoder-decoder that writes vision answers as text, pretrained on FLD-5B from a model-driven data engine. - [MM-SafetyBench](/papers/2311.17600): 1,680 harmful questions rendered as images in three ways, showing that pictures can bypass text-level refusals. - [PixelLM: Pixel Reasoning with Large Multimodal Model](/papers/2312.02228): an LLM that writes codebook tokens for each target, with a light pixel decoder that turns them into masks, trained on the MUSE dataset. - [Gemini: A Family of Highly Capable Multimodal Models](/papers/2312.11805): a family of decoders trained natively on text, image, audio and video, with a benchmark-heavy report and few reproducible details. - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](/papers/2403.05530): a sparse mixture-of-experts Pro and a distilled dense Flash, tested on needle recall and long-context QA to millions of tokens; closed, and light on architecture and training detail. - [FastV](/papers/2403.06764): drops low-attention image tokens after an early layer, with no training. - [Visual-RFT: Visual Reinforcement Fine-Tuning](/papers/2503.01785): reinforcement fine-tuning of Qwen2-VL with rule-based rewards for detection and classification. - [SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis](/papers/2503.10265): a zero-shot multi-agent workflow with a knowledge check and retrieval, evaluated on a surgical frame benchmark. - [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](/papers/2504.06958): Qwen2.5-VL-7B trained with GRPO on temporal grounding, tracking and grounded QA. - [InfoSeek](/papers/2302.11713): a visual QA benchmark for questions that need outside knowledge about the pictured entity. - [On the Adversarial Robustness of Multi-Modal Foundation Models](/papers/2308.10741): white-box APGD attacks with imperceptible perturbations that steer OpenFlamingo's captions and answers. - [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](/papers/2311.05437): a LLaVA-style model that writes tool calls in three fields and gets tool outputs back in the chat. - [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](/papers/2312.00784): draws boxes, masks, arrows and scribbles onto the image before a LLaVA-style encoder reads it. - [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](/papers/2312.09245): an MLLM that outputs one speed and one path decision for a motion planner, tested in CARLA. - [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](/papers/2312.16217): a LLaMA-Adapter fine-tuned to output a contact pixel and gripper directions for simulated pulling tasks. - [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](/papers/2504.17432): UniME, a two-stage recipe that turns an MLLM into an image-text embedder. - [Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models](/papers/2508.07173): 972 harmful seeds rendered in 24 text, image, video and audio combinations, with understanding-aware safety scores. - [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](/papers/2601.14724): a training-free KV-cache policy for streaming video that assigns a memory role to each layer. - [Sensory PID](/papers/2606.00959): a partial-information split of how video and audio shape omni-modal answers, and PID-based sample weights for vision-language fine-tuning. - [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](/papers/2202.03052): one encoder-decoder that writes text, image codes and box tokens from an instruction, with no task heads. - [PaLI: A Jointly-Scaled Multilingual Language-Image Model](/papers/2209.06794): a ViT and an mT5 encoder-decoder trained on a multilingual mix of image-text tasks, with the largest model scaling the ViT to 4B parameters. - [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](/papers/2209.09513): ScienceQA, 21,208 science questions with lectures and explanations, and answer-first chain-of-thought prompting. - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](/papers/2301.12597): a 188M Q-Former trained in two stages between a frozen image encoder and a frozen LLM. - [Multimodal Chain-of-Thought Reasoning in Language Models](/papers/2302.00923): two T5 models, one writing a rationale from text and frozen ViT features, the other answering from the text and the rationale. - [Visual Instruction Tuning](/papers/2304.08485): LLaVA, a frozen CLIP encoder joined to Vicuna by one trainable projection, tuned on 158K GPT-4-written instruction samples. - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](/papers/2304.10592): a frozen BLIP-2 vision stack and frozen Vicuna joined by one trainable projection, with a second stage on curated detailed descriptions. - [A Survey on Multimodal Large Language Models](/papers/2306.13549): a survey of the encoder, connector and LLM design, the three training stages and hallucination; it runs no experiments. - [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](/papers/2306.15195): writes image coordinates as plain numbers in the text, with a frozen CLIP encoder and one linear layer into Vicuna. - [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](/papers/2311.17005): 20 temporal video tasks in multiple-choice form, and VideoChat2, a three-stage video model evaluated on them. - [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](/papers/2404.14219): the phi-3 family of small models trained on filtered and synthetic data, including the 4.2B vision model phi-3.5-Vision. - [GPT-4o System Card](/papers/2410.21276): a closed omni-model's system card, covering speech safety, preparedness scores and third-party tests; no code or weights. - [MedGemma 1.5 Technical Report](/papers/2604.05081): MedGemma 1.5 4B adds 3D CT and MRI slice sequences, whole-slide tiles, X-ray localisation and longitudinal chest X-ray comparison. - [ClipCap: CLIP Prefix for Image Captioning](/papers/2111.09734): a frozen CLIP embedding mapped to a GPT-2 prefix, with an MLP or transformer mapper trained on captions. - [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](/papers/2305.11000): speech units added to a LLaMA vocabulary, three training stages, and human-judged examples only. - [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](/papers/2311.16922): a training-free sampler that contrasts the logits for the image with those for a noised copy. - [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](/papers/2305.07895): a 1000-item test of text reading in multimodal models across five jobs, scored by checking for the answer string. - [GLaMM: Pixel Grounding Large Multimodal Model](/papers/2311.03356): a chat model that attaches a mask to each object phrase it names, with the GCG benchmark and the GranD dataset. - [Generative Multimodal Models are In-Context Learners](/papers/2312.13286): Emu2, a 37B model that predicts text tokens or image embeddings in one interleaved sequence, with few-shot results and an image decoder. - [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](/papers/2602.01990): mixture of LoRA experts for LLaVA, trained one task at a time without replay, with router and expert stabilisation. - [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](/papers/2304.14178): a CLIP encoder and a learnable visual abstractor on LLaMA-7B, with a frozen-LLM stage and a LoRA stage. - [Otter: A Multi-Modal Model with In-Context Instruction Tuning](/papers/2305.03726): OpenFlamingo-9B tuned on MIMIC-IT with in-context image and instruction examples, training only the resampler and gated attention. - [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](/papers/2305.11175): a language-guided tokenizer and a LoRA-tuned Alpaca that writes detection and segmentation as class and location tokens. - [MotionGPT: Human Motion as a Foreign Language](/papers/2306.14795): VQ-VAE motion codes added to a Flan-T5 vocabulary, with text-to-motion, motion-to-text, prediction and in-between tasks. - [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209): the MMVP benchmark of CLIP-blind image pairs, and interleaved CLIP and DINOv2 tokens for LLaVA. - [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](/papers/2412.14171): VSI-Bench of spatial questions over indoor video, with text-only reasoning and cognitive-map probes. - [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](/papers/2311.15826): a LLaVA-1.5-style remote-sensing chat model with boxes written as text and task tokens, trained on a dataset built from aerial detection labels. - [Think Then Embed: Generative Context Improves Multimodal Embedding](/papers/2510.05014): a multimodal LLM writes a short reasoning text before the embedding is computed, in teacher, small-reasoner and unified variants. - [FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction](/papers/2509.04018): a VLA policy with a Qwen2.5-VL supervisor that is queried at gripper changes, plus a fusion of past action predictions. - [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](/papers/2512.08186): DualVLN, a Qwen2.5-VL planner that outputs pixel goals and a diffusion transformer that outputs 32 waypoints, run asynchronously. - [End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering](/papers/2411.05755): VLMnav, a zero-shot navigation policy in which a VLM picks one of several numbered arrows drawn on a depth-checked camera image. - [Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs](/papers/2411.14256): a small imitation driver that a frozen multimodal LLM steers with LEFT, MIDDLE or RIGHT hints, tested on a real RC car. - [InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models](/papers/2411.11394): route instructions written by a large multimodal model from house-tour videos, checked for consistency, used to pretrain a navigation agent. - [Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces](/papers/2501.09024): a LoRA-tuned LLaVA-1.5-7B on a human-annotated question set about robot-human scenes, rated above two prompted models by human judges. - [ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model](/papers/2304.03086): a review of dental uses for multi-modal LLMs, with one illustrative pipeline and no evaluation. - [Capabilities of Gemini Models in Medicine](/papers/2404.18416): Med-Gemini, Gemini 1.0 and 1.5 models tuned for medicine, with self-trained search, multimodal tuning and long-context prompting; the models are not released. - [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](/papers/2406.01584): a LLaMA2-7B VLM that reads mask-pooled region vectors and an optional depth input, trained on OpenImages questions built from 3D scene graphs. - [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](/papers/2411.16537): about 3M synthetic spatial questions from 3D scans, used to fine-tune 2D and 3D VLMs, with a non-commercial official generator.