Multimodal language model

Overview

A multimodal language model takes a non-text input, such as an image, a video, an audio clip or a depth map, next to text, and produces an answer. Three choices separate the designs.

1. How the non-text input reaches the model. Through a small trained bridge into a frozen LLM, through a shared embedding space from a frozen encoder, through the same token vocabulary as text, or through one network trained from the start on all modalities.
2. Which parts train. Only the bridge, the bridge plus adapters, or every weight.
3. What the output is. Text only, or text plus structured outputs such as boxes, masks and depth, written as tokens or produced by an extra decoder.

A survey of the field describes the same parts as a modality encoder, a pre-trained LLM and a connector between them 2306.13549 §2. Most designs below differ only in what the connector is and which parts it lets train.

The bridge designs are the most common in the cards. InstructBLIP keeps both its image encoder and its LLM frozen and trains only a Q-Former 2305.06500 §2.2 2305.06500 §2.3. Video-ChatGPT and PandaGPT also keep the LLM frozen. Each trains a linear map into the LLM's input space, and PandaGPT adds LoRA adapters 2306.05424 §3.1 2305.16355 §3. At the other end, Med-PaLM M updates every weight of a multimodal base for one domain 2307.14334 §4.2. Gemini 1.0 is trained as a multimodal decoder from the start, and the report does not describe its visual encoder or bridge 2312.11805 §2.

The same family also writes tool calls, produces embeddings, picks robot or driving actions, and is tested for safety and for which input it relies on. Those uses have their own sections below.

Variants

Frozen LLM with an instruction-aware bridge

Used by InstructBLIP. The image encoder is a frozen ViT-g/14, and the LLM is a frozen FlanT5 or Vicuna model [arxiv:2305.06500 §2.6]. The Q-Former's learned queries attend to the image features, and the instruction tokens also go into the Q-Former. That makes the visual features depend on the task being asked 2305.06500 §2.3. Removing this instruction-aware step lowers the held-in average for FlanT5-XL from 94.1 to 89.8 2305.06500 tab. 2. For video, the paper samples four frames, runs each through the encoder and Q-Former separately, and concatenates the outputs, so the LLM receives four times as many visual vectors 2305.06500 §2.5.

Only the Q-Former is trained. The paper reports 188M trainable parameters in finetuning, against 1.2B for BLIP-2 finetuning 2305.06500 §3.5.

Frozen LLM with a linear bridge over video

Used by Video-ChatGPT. A frozen CLIP ViT-L/14 encodes each frame, and the patch features are pooled two ways: averaged over the grid to give one vector per frame, and averaged over frames to give one vector per patch position. The two sets are stacked, giving T+N tokens, and one linear layer maps them into the 7B Vicuna model's embedding space [arxiv:2306.05424 §3.1] 2306.05424 eq. 1 2306.05424 eq. 2. The bridge gets no instruction input, so the visual tokens do not depend on the question. The paper does not test whether that matters.

Only the linear layer is trained, and the paper reports about 3 hours on 8 A100 40GB GPUs 2306.05424 §5.1. The paper states no frame count. Its released code samples up to 100 frames at fixed positions, so frame handling has to come from the code if you want to match the released weights (see the card's Paper vs code).

Frozen LLM with a query bridge and no instructions

Used by BLIP-2. A frozen image encoder and a frozen LLM are joined by a Querying Transformer with 32 learnable queries of width 768, initialised from BERT-base 2301.12597 §3.1. The Q-Former is the only part that is trained in the first stage. It learns three losses at once: image-text contrastive, image-grounded text generation and image-text matching, each with its own attention mask 2301.12597 §3.2. In the second stage, a fully connected layer maps the query outputs into the LLM's embedding width, and the result is prepended to the text as soft visual prompts 2301.12597 §3.3. The paper reports 188M trainable parameters for the Q-Former 2301.12597 tab. 1, but other tables give 103M to 108M for the same component 2301.12597 tab. 2. The paper says the first stage matters: without it, zero-shot VQA is much lower for both LLM types 2301.12597 §4.1.

Zero-shot VQAv2 test-dev reaches 65.0 with a ViT-g encoder and FlanT5-XXL, against 56.3 for Flamingo-80B 2301.12597 tab. 2. The paper's "8.7%" is a difference of percentage points, not a relative gain 2301.12597 §4.1. The paper's zero-shot results are for this design alone; instruction tuning of the bridge is the InstructBLIP variant above.

Frozen encoder, one linear projection, two-stage tuning

Used by LLaVA and MiniGPT-4. LLaVA takes features from a frozen CLIP ViT-L/14, uses the layer before the last, and maps them through one trainable matrix into Vicuna 2304.08485 §4.1 2304.08485 tab. 8. Stage 1 trains only that matrix on about 595K caption pairs. Stage 2 trains the matrix and the LLM on 158K GPT-4-written instruction samples, with the image encoder still frozen 2304.08485 §4.2. The authors report 85.1 on their COCO benchmark, scored by a text-only GPT-4 judge against a reference answer 2304.08485 tab. 4. On ScienceQA, LLaVA reaches 90.92 on the test split 2304.08485 tab. 7, and the authors list "MM-CoT Large" at 91.68, a number that differs from the one in its own paper (see Contradictions).

MiniGPT-4 freezes a BLIP-2 vision stack and Vicuna, and trains one linear layer 2304.10592 §3. Stage 1 aligns that layer on about 5M image-caption pairs. The authors say caption data alone gives unnatural output, with repeated words and fragments 2304.10592 §3.1. Stage 2 fine-tunes on about 3,500 detailed descriptions that were produced by the stage-1 model and then edited by ChatGPT and by hand 2304.10592 §3.2. The paper reports 400 stage-2 steps at batch 12 2304.10592 §3.3, but the released config runs more steps (see the card's Paper vs code). On a 100-image set, human judges rate 65 of 100 requests satisfied against 5 for BLIP-2, and the failure rate of detailed descriptions falls from 35% to 2% after stage 2 2304.10592 tab. 1 2304.10592 tab. 3. The paper puts stage 1 at about ten hours on four A100 GPUs and stage 2 at about seven minutes on one 2304.10592 §3.1 2304.10592 §3.3.

Learned visual abstractor, then LoRA on a language model

Used by mPLUG-Owl. A CLIP ViT-L/14 encoder feeds a learnable visual abstractor, which compresses the patch features into a short set of visual tokens for LLaMA-7B 2304.14178 §3.1. The paper does not state how many visual tokens the abstractor produces. Stage 1 trains the encoder and the abstractor on image-caption pairs while the LLM stays frozen, with the loss on text tokens only 2304.14178 §3.2. Stage 2 freezes the whole model and trains LoRA adapters on the LLM, using text-only and image-text instructions 2304.14178 §3.2. The authors report 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4. The rating is manual, and the paper does not say how many raters scored the answers 2304.14178 §4.2.

The stage-1 settings conflict inside the paper. The text gives learning rate 0.0001 and 2,000 warmup steps 2304.14178 §4.1, and the appendix table gives 2e-4 and 375 steps 2304.14178 app. A. The released repository contains only the stage-2 code, so stage 1 cannot be checked (see the card's Pitfalls).

Frozen LLM with in-context examples and gated cross-attention

Used by Otter. The base is OpenFlamingo-9B, with a frozen CLIP ViT-L/14 encoder and a frozen MPT-7B decoder. A perceiver resampler and gated cross-attention layers inside the decoder are the only trained parts 2305.03726 §3.1. The paper puts this at about 1.4B parameters 2305.03726 §3.1.

The difference from the bridges above is the training data. Each MIMIC-IT item holds a query and a set of in-context examples, which are other image-instruction-response sets shown before the query 2305.03726 §4. For the LLaVA-derived subsets, the examples are the most similar training items by CLIP image similarity or instruction text similarity 2305.03726 §4.1. The MIMIC-IT dataset has about 3M instances and 2.2M unique instructions 2305.03726 tab. 1. The paper reports that training with in-context examples converges faster and does better at few-shot evaluation. On MME cognition with two-shot evaluation, a model trained without them scores 1,310.7 and one trained with two-shot examples scores 1,430.3 2305.03726 tab. 10. The runs are single, and the paper reports no variance. The video results use a ChatGPT-based judge 2305.03726 §6.2.

Frozen CLIP prefix and a small mapper for captions

Used by ClipCap. A frozen CLIP image encoder gives one embedding, and a mapping network turns it into k vectors that sit in front of the caption tokens of GPT-2 2111.09734 §3.1 2111.09734 eq. 3 2111.09734 eq. 4. The method needs only images and captions 2111.09734 §1. The paper has two variants. One fine-tunes GPT-2 with a single-hidden-layer MLP mapper, 156M parameters in total. The other freezes GPT-2 and trains a transformer mapper, 43M parameters 2111.09734 §3.2 2111.09734 tab. 1. On Conceptual Captions, the MLP variant reaches 87.26 CIDEr against 77.57 for VLP, and the transformer variant reaches 71.82 2111.09734 tab. 1 (A). On COCO, the transformer variant reaches 113.08 CIDEr against 124.12 for Oscar, which also uses object tags 2111.09734 tab. 1 (C) 2111.09734 §4.

The transformer variant trains for 6 hours on one GTX 1080, with the prefix set to 10 vectors 2111.09734 tab. 1 2111.09734 §4. The released README uses a prefix of 40 for that variant (see the card's Paper vs code).

One embedding space for many modalities

Used by PandaGPT. A frozen ImageBind encoder maps image, video, audio, depth, thermal and IMU inputs into one embedding space. One trainable linear projection maps the resulting vector into a frozen Vicuna-13B, and LoRA adapters on the attention modules are the only other trainable weights 2305.16355 §3. Training uses 160k image-language examples for two epochs 2305.16355 §3. Other modalities work without their own training because the encoder already places them in the shared space [arxiv:2305.16355 §1]. Several inputs are combined by adding their vectors 2305.16355 §4.

The paper reports example conversations and no benchmark numbers, so its cross-modal results are demonstrations, not measurements 2305.16355 §4. The released data and delta weights carry non-commercial terms, so a commercial build cannot start from them (see the card's Pitfalls).

Natively multimodal decoder

Used by Gemini 1.0. A decoder-only Transformer with a 32,768-token context, trained on text, images, audio and video [arxiv:2312.11805 §2] 2312.11805 tab. 1. Audio enters as 16 kHz Universal Speech Model features rather than as a transcript 2312.11805 §2. Video enters as frames in the context; the video benchmarks use 16 frames per clip 2312.11805 §5.2.2. The model can also emit images as discrete tokens, though the image decoder is not described 2312.11805 §2.

The family has four sizes: Ultra and Pro, whose parameter counts are not stated, and Nano-1 (1.8B) and Nano-2 (3.25B), which are distilled from larger Gemini models and 4-bit quantized 2312.11805 tab. 1. The quantization method and distillation objective are not stated 2312.11805 §2.

Post-training follows a pipeline of prompts, supervised fine-tuning, a reward model trained on human preferences, and iterated RLHF 2312.11805 §6.3. The report gives no RL algorithm, KL penalty, learning rate or data sizes for it 2312.11805 §6.3.

Sparse mixture-of-experts decoder for million-token context

Used by Gemini 1.5. Pro is a sparse mixture-of-experts Transformer: a learned router sends each input to part of the parameters, so the total can grow while the work per input stays fixed [arxiv:2403.05530 §3.1]. The authors say architecture changes let inputs reach 10 million tokens without losing quality, but do not describe those changes [arxiv:2403.05530 §3.1]. Flash is a dense Transformer with the same interface. It is distilled online from Pro, and its optimiser is named only as higher-order and preconditioned 2403.05530 §3.2. The parameter count, the training data mixture beyond its domains and the training hyperparameters are not stated 2403.05530 §4 2403.05530 app..

Fully fine-tuned generalist from a multimodal base

Used by Med-PaLM M, built on PaLM-E at 12B, 84B and 562B parameters. PaLM-E's language model is a decoder-only Transformer, and its image encoder is a ViT with 4B parameters for the 12B model and 22B for the 84B and 562B models 2307.14334 §4.1. Each 224 x 224 image becomes 256 tokens 2307.14334 app. A.2.2, and each fine-tuning input holds at most one image 2307.14334 §4.2.

Unlike InstructBLIP, nothing is frozen. All parameters are updated on MultiMedBench, with instruction-style prompts and text targets 2307.14334 §4.2. Learning rate is 5e-5 for the 12B and 84B models and 2.5e-5 for the 562B model, with a constant schedule and Adafactor 2307.14334 tab. A.2 2307.14334 §4.2. For most tasks, the exemplar's image is replaced by a text placeholder, so the prompt is text-only except for the final image 2307.14334 §4.2 2307.14334 fig. 2.

Gemini tuned for medicine, with self-trained search and long context

Used by Med-Gemini. The paper starts from Gemini 1.0 and 1.5 models and builds three tracks, each with its own tuned model. The text track, Med-Gemini-L 1.0 on Gemini 1.0 Ultra, is fine-tuned on reasoning chains the model writes itself. Each training question gets one chain without search results and one with them, and chains that end in a wrong answer are dropped 2404.18416 §2.1. At inference the model samples several reasoning paths and searches the web only when those paths disagree strongly, judged by the entropy of their answers. The entropy threshold is not stated 2404.18416 §2.1.

The image track, Med-Gemini-M 1.5 on Gemini 1.5 Pro, is instruction-tuned on eight medical multimodal tasks from six datasets. Each dataset's share of the mixture is roughly proportional to its size 2404.18416 §2.2 app. D.1. Med-Gemini-S 1.0 adds a cross-attention encoder for raw 12-lead ECG signals to Gemini 1.0 Nano 2404.18416 §2.2. The long-context track uses no new weights: a patient's notes go into one prompt for a two-step check, and a video goes in with a task instruction and no examples 2404.18416 §2.3.

The results are author-reported. On MedQA the text model reaches 91.1 against 90.2 for GPT-4 with MedPrompt 2404.18416 tab. 1. On ECG-QA the Nano-based model reaches 57.7 against 51.6 2404.18416 tab. 2. The results are not uniform: the EHR search F1 is 0.77 against 0.78 for a heuristic baseline 2404.18416 tab. 3, and surgical view classification is 55.2 against 67.0 for a ResNet3D the authors reproduced 2404.18416 tab. 3. The multimodal baselines are not the same in every row [arxiv:2404.18416 tab. 2]. The model weights and most code are not released, and the training recipe leaves out learning rates and step counts (see the card's Paper vs code).

Used by Unified-IO. Text is written with SentencePiece tokens, points and boxes with 1000 location tokens, and images or dense maps such as depth with the codes of a 16384-entry VQ-GAN trained on ImageNet 2206.08916 §3.1 2206.08916 §3.2. One T5-style encoder-decoder reads the prompt and the image patches and writes the answer as tokens, which are then converted back into a box, a mask, a depth map or a sentence 2206.08916 §3.1. The model is trained on 95 datasets with no task-specific heads 2206.08916 §3.3. Model sizes run from 71M to 2,925M parameters 2206.08916 tab. 2.

The XL model reports 64.3 on the GRIT test split against 32.0 for GPV-2 2206.08916 tab. 3. The authors say the ImageNet-only image tokenizer limits image generation and depth and normal quality 2206.08916 §4.6. The released code is inference-only, so the training recipe rests on the paper alone (see the card's Paper vs code).

OFA uses the same idea with one shared vocabulary: BPE text, 1,000 location tokens for box corners, and VQGAN-style image codes, so a 256 x 256 image becomes a 16 x 16 code sequence 2202.03052 §3.1. Every task, pretraining included, is generation from an instruction plus inputs 2202.03052 §3.2. The authors pretrain on public image-text pairs plus unimodal data; the abstract says 20M pairs, while the data table lists 15.25M captioning and matching samples 2202.03052 tab. 11. Reported results include 82.0 on VQA test-std and 154.9 CIDEr for COCO captions with CIDEr optimisation 2202.03052 tab. 2 2202.03052 tab. 3. The unimodal results are not always the best, and the paper's zero-shot results are sensitive to the wording of the instruction 2202.03052 §4.3.

Dental image, voice and records in one LLM, described but not built

Proposed in a review, not built or measured. The authors sketch a system in which three inputs, a dental X-ray or cone-beam CT, a patient's voice and old records, feed one LLM that proposes a diagnosis and a treatment 2304.03086 fig. 11. The voice is converted to text and plotted as a waveform and spectrogram, and the results go to the same model [arxiv:2304.03086 p. 9]. The paper names no model for the image step, gives no training data and reports no evaluation. Each example is one output the authors chose 2304.03086 fig. 1-12. Use it as a list of possible inputs and risks, not as a design. Its own caries example misses a finding the authors note themselves 2304.03086 p. 10. Card: ChatGPT for Shaping the Future of Dentistry.

Vision-centric outputs written as class and location tokens

Used by VisionLLM. A language-guided tokenizer reads the image at four scales, cross-attends to BERT features of the prompt, and turns them into M = 100 learned image tokens with a Deformable-DETR-style transformer 2305.11175 §3.3. An Alpaca-7B decoder with LoRA then writes the answer. The vocabulary gains 1,025 location tokens, covering offsets of -512 to 512 from a reference point, and 512 class tokens 2305.11175 §3.4. The instruction names the output format, and the decoder reads those format queries first, so the outputs come out in parallel rather than one token at a time 2305.11175 §3.4.

Object detection and instance segmentation use the same output. A box is two corner points, and a mask is a polygon of boundary points: more than eight in the method, and 8 to 24 in the experiments 2305.11175 §3.2 2305.11175 §4.2. Training uses a focal loss on the tokenizer queries and bipartite matching for set outputs 2305.11175 app. B. Its reported box AP is 44.6 with ResNet-50 and 60.2 with InternImage-H 2305.11175 tab. 1. The number of location bins matters: 257 bins give 34.9 AP and 1,025 give 44.8 in the ablation 2305.11175 tab. 3c.

The linked repository has no implementation of this method at the pinned commit (see the card's Paper vs code), so an implementation rests on the paper alone.

Continuous image embeddings in one autoregressive stream

Used by Emu2. Each image becomes 64 embeddings: patch features from a visual encoder are average-pooled to an 8 x 8 grid, then mapped by one linear projection 2312.13286 §2.1. The embeddings sit between image markers in an interleaved sequence with text. One autoregressive model of 37B parameters predicts the next text token, with cross-entropy, or the next image embedding, with a regression loss. The paper gives no weighting between the two terms 2312.13286 §2.2.2. Because image outputs are continuous vectors rather than vocabulary entries, a separate SDXL-based diffusion decoder turns predicted embeddings into pictures 2312.13286 §2.2.3. The language-model pretraining runs in two stages, the second with the visual encoder frozen 2312.13286 §2.2.2.

The few-shot results rise with the number of demonstrations. VQAv2 goes from 33.5 at zero shots to 67.0 at four and 68.8 at sixteen 2312.13286 tab. 1. The instruction-tuned Emu2-Chat reaches 84.9 on VQAv2 2312.13286 tab. 2. The zero-shot setting needs a particular prompt: the plain one made the model write sentences, so the authors used a modified one 2312.13286 app. E.1. The released code is inference-only, and its image decoding runs the full model once per output slot (see the card's Pitfalls).

Discrete speech units as vocabulary tokens

Used by SpeechGPT. A self-supervised speech encoder gives frame features, k-means assigns each frame to a cluster, and adjacent repeats are removed to give a unit sequence 2305.11000 §4.1. The paper does not state the number of clusters. The unit indices are added to the LLM's vocabulary, so speech in and speech out are ordinary tokens, with no recogniser or synthesiser in the loop 2305.11000 §4.2 2305.11000 §1. Removing repeats also removes durations, so the model does not control timing or emphasis 2305.11000 §3.1.

Training has three stages 2305.11000 §4.2. Unit pre-training uses about 60K hours of English audiobook speech 2305.11000 §5.1. Instruction tuning uses about 9 million speech-text pairs from English ASR corpora, turned into recognition and synthesis tasks 2305.11000 §3.1. A LoRA stage then tunes on chain-of-modality data, where a text answer is followed by a speech answer, with rank 8 and alpha 16 2305.11000 §3.2 [arxiv:2305.11000 app. D]. The paper's backbone is LLaMA-13B, while the released scripts use 7B (see the card's Paper vs code).

The evidence is qualitative: a few transcription and read-aloud examples, and ten spoken dialogues judged by people. No metric and no baseline are reported 2305.11000 §5.2 2305.11000 tab. 1 2305.11000 tab. 2. The paper lists three limits: no control over emotion or tone, a text answer written before the speech answer, and no multi-turn support [arxiv:2305.11000 §6].

Human motion codes as vocabulary tokens

Used by MotionGPT. The same idea as speech units, applied to 3D human motion rather than audio. A VQ-VAE turns a motion clip into codes from a codebook of 512 entries, after a temporal downsampling of four 2306.14795 §3.1 2306.14795 §4.1. The codes become words in a Flan-T5 vocabulary, wrapped in start and end tokens 2306.14795 §3.2. The base model is Flan-T5 at 220M parameters 2306.14795 §4.1.

Training has three stages. The tokenizer is trained first and then frozen. The language model is pretrained on span corruption over text and motion, and on supervised motion-text pairs from HumanML3D and KIT 2306.14795 §3.3. It is then instruction-tuned on prompt templates, with more than a thousand prompts across about 15 tasks 2306.14795 §3.3 2306.14795 tab. 13. One model covers text-to-motion, motion-to-text, prediction and in-between completion 2306.14795 §4.

On HumanML3D, the task-tuned model reaches FID 0.232 for text-to-motion, while T2M-GPT reports 0.116 2306.14795 tab. 3. For motion-to-text, BLEU@4 is 12.47 against 7.00 for TM2T 2306.14795 tab. 4. The abstract calls the results state of the art, and the text of §4.2 says "competitive" on most metrics 2306.14795 §1 2306.14795 §4.2.

Coordinates as plain text

Used by Shikra. Shikra keeps the usual bridge: a frozen CLIP ViT-L/14, one trainable linear layer giving 256 visual tokens, and Vicuna at 7B or 13B 2306.15195 §5.1. It adds no coordinate vocabulary. A box is written as [x_min,y_min,x_max,y_max] and a point as [x_center,y_center], normalised to the image size and kept to three decimals, and it is tokenised like any other text 2306.15195 §5.2. Inputs can contain coordinates as well as outputs.

In a toy comparison trained only on referring-expression data, models that write numbers beat models with extra coordinate tokens on every split 2306.15195 tab. 2. The cost is length: numbers take more tokens per box, which matters in dense scenes 2306.15195 §6.2. Shikra-7B reaches 87.01 on RefCOCO val, a generalist result, against 90.56 for the specialist G-DINO-L 2306.15195 tab. 3. The official code is CC BY-NC 4.0, so an implementation should not copy from it.

Remote-sensing chat with boxes written as text

Used by GeoChat. The design follows LLaVA-1.5. A CLIP ViT-L/14 has its position embeddings interpolated to take 504 x 504 images, which gives 1296 patch tokens. A two-layer MLP maps them into a Vicuna-v1.5 7B model that is tuned with LoRA, while the encoder and the MLP stay frozen 2311.15826 §3.1 2311.15826 §3.2. Three additions set it apart. Boxes are plain text: four coordinates scaled to [0, 100] and a rotation angle, and region boxes can appear in the prompt as well as in the answer 2311.15826 §3.1. Three task tokens select the answer style: grounding, describing the object in a given box, and locating a named phrase 2311.15826 tab. 1. The training pairs are written by a text-only model from detection labels over aerial images, about 306k in the training split 2311.15826 §4 2311.15826 tab. 1.

On zero-shot scene classification, the authors report 84.43% on UCMerced and 72.03% on AID, against 68.00% and 51.00% for LLaVA-1.5 [arxiv:2311.15826 tab. 5]. Their grounding benchmark is built with the same pipeline as the training data, so it is not an independent test 2311.15826 §5.4. The paper's LoRA targets differ from the released code (see the card's Paper vs code).

Prompt-driven encoder-decoder

Used by Florence-2. One sequence-to-sequence model takes an image and a text prompt and generates the response as text. The tokenizer vocabulary is extended with 1,000 quantized coordinate bins. Boxes are written as two corners, text regions as four corners and referring-segmentation masks as polygons, all as sequences of these bins 2311.06242 §3.

A DaViT backbone turns the image into visual tokens. A linear projection and LayerNorm map them to the decoder width, and they are placed before the prompt embeddings to form the encoder input 2311.06242 §3. Every task uses the same cross-entropy loss over the target tokens 2311.06242 eq. 1 2311.06242 §3.

The paper reports two sizes: base at 0.23B and large at 0.77B parameters 2311.06242 tab. 4. Zero-shot, the large model reaches 135.6 CIDEr on COCO captions and 84.4 R@1 on Flickr30k grounding 2311.06242 tab. 4. The baselines, Flamingo 80B and Kosmos-2 1.6B, are as the authors report them 2311.06242 §6.2.

Pretrained ViT and multilingual encoder-decoder, scaled together

Used by PaLI. A plain Vision Transformer's patch features, without pooling, feed an mT5 encoder-decoder, and the prompt selects the task. Every task is image and text in, text out, with no task heads [arxiv:2209.06794 §3.1]. The authors argue that earlier large models scaled the language side far more than the vision side, so PaLI-17B pairs a 4B-parameter ViT with mT5-XXL 2209.06794 app. A.1. Pretraining mixes eight tasks over WebLI, a corpus of about 10B images and 12B alt-texts in 109 languages, of which the top 10% by cross-modal similarity are kept 2209.06794 §3.2 app. B. The corpus is not released, and the largest run used 1,024 TPUv4 chips for 7 days 2209.06794 app. A.1. Results include 149.1 CIDEr on COCO Karpathy test 2209.06794 tab. 1, 84.3 on VQAv2 test-dev 2209.06794 tab. 3, and 53.6 average CIDEr over 35 languages on Crossmodal-3600, against 47.0 for the 3B model 2209.06794 tab. 2.

The paper's text says the visual tokens go to the encoder and also reach the decoder through cross-attention, without saying how they combine with the text 2209.06794 §3.1. Treat that interface as open. Since the largest run is so costly, reproduce a smaller variant first.

Small language model with a vision adapter

Used by Phi-3.5-Vision. A CLIP ViT-L/14 encoder feeds phi-3.5-mini (4.2B in total), with visual and text tokens interleaved and high-resolution images split into a grid of blocks 2404.14219 §7.1. Pretraining uses 0.5T tokens of interleaved image-text documents, pairs, OCR PDF data and charts, with the loss on text tokens only 2404.14219 §7.1. The authors report 91.3 on ScienceQA test and 43.0 on MMMU validation 2404.14219 tab. 5. The report leaves out learning rates, batch sizes and data mixture proportions 2404.14219 §2, so the training recipe cannot be rebuilt from it. The vision evaluations also use settings that differ from published pipelines 2404.14219 §7.2.2, and the authors say the model sometimes fails to refuse harmful requests that depend on OCR, such as scam images 2404.14219 §7.4.

Language model with a pixel decoder for masks

Used by PixelLM. A LLaVA-based model, LoRA-tuned, writes an answer that contains a group of learned codebook tokens for each target. The hidden states of those tokens, together with CLIP features from two layers, go into a small pixel decoder that produces one mask per target, so no separate SAM model is needed 2312.02228 §3.1. The method is trained on MUSE, a multi-target dataset whose answers are generated with GPT-4V [arxiv:2312.02228 §4.2].

On the paper's MUSE benchmark, the test-overall gIoU and cIoU are 39.2 and 46.3 for PixelLM-7B, against 12.8 and 27.1 for LISA-7B. The paper's TFLOPs estimate is 3.57 against 7.16 2312.02228 tab. 1. On refCOCO val, LISA scores higher than PixelLM 2312.02228 tab. 3. These are the authors' numbers, and the released code does not train the target refinement loss the paper describes (see the card's Paper vs code).

Pixel-grounded chat with a mask per named object

Used by GLaMM. A global CLIP encoder gives scene features that are projected into a Vicuna-7B model with LoRA. A region encoder pools each user box into one vector, which takes the place of a box token in the prompt 2311.03356 §3.1. The LLM writes a <SEG> token wherever it names an object. The hidden state at each <SEG> goes through a projection into a SAM-style pixel decoder, which returns one mask per token, with SAM's image encoder frozen 2311.03356 §3.1 2311.03356 app. A.2. The answer format wraps each grounded phrase in <p> tags followed by <SEG> 2311.03356 §3.2. Training pretrains on GranD for four tasks at once and then finetunes per task 2311.03356 app. A.2 2311.03356 app. A.3.

On the GCG benchmark, the test AP50 is 29.2 and the test recall 40.8, against 24.8 and 35.5 for LISA, which the authors adapted to the task 2311.03356 tab. 3. On referring-expression segmentation, GLaMM scores 79.5 on refCOCO val against 74.9 for LISA-7B, but the table does not name its metric 2311.03356 tab. 4. The paper names CLIP ViT-H/14 for the global encoder, while the released code defaults to a smaller model, so the released weights do not match the stated encoder (see the card's Paper vs code). GranD_f repurposes public referring datasets with GPT-4-written captions 2311.03356 §4.5, and the card did not clear those terms for commercial use (see the card's Pitfalls).

Region vectors in the prompt, with an optional depth connector

Used by SpatialRGPT. The model is a LLaVA-style VLM on LLaMA2-7B. The image is encoded once, two deconvolution layers upsample the feature map, and the features inside each region mask are averaged into one vector per region 2406.01584 app. G.2. A linear connector maps each region vector into the language space, and it is placed where a region placeholder appears in the text 2406.01584 §3.3. Training samples either the box or the mask of a region at random, and inference uses the mask when one is given, or converts the box to a mask with SAM [arxiv:2406.01584 §3.4].

Relative depth from an off-the-shelf model goes through the same visual encoder, then a separate depth connector, and is averaged per region in the same way. Depth is an extra input: the model also runs without it 2406.01584 §3.3. The training data is synthetic. Objects in about 1M OpenImages photos are detected and given metric depth and camera calibration, placed as boxes in a 3D scene graph, and then turned into questions about size, left and right, above and below, and distance 2406.01584 §3.1 2406.01584 §3.2. The authors report 5M regions and 8M template questions 2406.01584 §3.2.

The paper's stage 2 freezes the RGB connector and the LLM, and trains only the depth connector. The released stage-2 script trains the LLM and projector too (see the card's Paper vs code).

On the authors' SpatialRGPT-Bench, the 7B model averages 91.78 in qualitative accuracy, against 58.14 for GPT-4V 2406.01584 tab. 1. On BLINK relative depth it reaches 82.3, against 66.9 for GPT-4V-Turbo 2406.01584 tab. 4. GPT-4 judges the answers, so the scores depend on that judge 2406.01584 app. J.

Visual prompts drawn onto the image

Used by ViP-LLaVA. Boxes, ellipses, points, triangles, masks, mask contours, arrows and scribbles are alpha-blended into the RGB image, and only the pixels under a prompt change 2312.00784 §3.1 [arxiv:2312.00784 eq. 1]. The composite goes through a frozen CLIP ViT-L/14 at 336 px. The paper keeps one early layer and four deeper layers, concatenates them, and maps them with a LayerNorm and a two-layer MLP 2312.00784 §3.1 [arxiv:2312.00784 §5.1]. No region encoder is added. In the authors' ablation, the multi-layer features raise Visual7W from 82.44 to 86.04 and PointQA from 70.23 to 70.86 for the 7B base 2312.00784 §5.4.

Training uses 520k region-prompt pairs built on existing region-annotated datasets 2312.00784 §3.2. The paper's alpha range and layer indices differ from the released code, which matters if you want the released weights to behave the same (see the card's Paper vs code).

Numbered arrows drawn on the image, chosen zero-shot

Used by VLMnav. No model is trained. A depth-based test finds directions the robot can drive in without a collision, and each one becomes a candidate action 2411.05755 §3.1. The candidates are drawn on the camera image as numbered arrows, with a turn-around option labelled 0 2411.05755 §3.3. One call asks the VLM for a single number, and a second call on the un-annotated image decides when to stop 2411.05755 §3.4 2411.05755 §3.5.

On Habitat ObjectNav, the authors report 50.4% success and 0.210 SPL with Gemini Flash, against 24.6% and 0.106 for PIVOT, which samples its actions from a Gaussian 2411.05755 tab. 1 2411.05755 §2. Replacing the depth-based arrows with static, evenly spaced ones gives 33.2% success, and turning off sliding along walls gives 12.9% 2411.05755 tab. 1. The method assumes ground-truth depth and pose from the simulator 2411.05755 §4, so the figures do not show how it would fare on a real robot.

Tool-calling planner

Used by LLaVA-Plus. Each assistant turn has three fields: thoughts, a JSON list of actions with API_name and API_params, and the reply value. An empty action list means no tool is called [arxiv:2311.05437 §2.1]. A skill-oriented sample has four turns: the question, the tool call, the tool output written as text, and the answer 2311.05437 eq. 2 2311.05437 §2.2. The skills cover detection, segmentation, tagging, captioning, OCR, retrieval, generation and editing 2311.05437 tab. 2.

The 7B model scores 69.5 on LLaVA-Bench In-the-Wild against 57.1 for LLaVA 2311.05437 tab. 4. The best-scoring setting, "All Tools", writes tool outputs into the context before answering, so it does not test tool choice. The choice evidence is the "Fly" setting, which scores 59.1 with thoughts and 48.7 without them 2311.05437 tab. 4.

Embedder built from a multimodal LLM

Used by UniME. The embedding is the normalised hidden state of the last token 2504.17432 §3.2. Stage 1 trains only the LLM, with LoRA and text alone, so that its batch similarities copy those of a text teacher, NV-Embed V2, whose embeddings are computed offline [arxiv:2504.17432 §3.2] 2504.17432 §3.4. Stage 2 trains on image-text pairs with InfoNCE over the positive and eight hard negatives, after removing candidates that score above the positive plus a margin of 0.1 2504.17432 §3.3 [arxiv:2504.17432 §4.1].

On MMEB, the stage-1 model alone reaches 40.3 and stage 2 alone 63.8 with Phi-3.5-V, and both together reach 64.2 2504.17432 tab. 3. The authors report 70.7 overall with LLaVA-OneVision-7B and 66.6 with LLaVA-1.6, against 63.3 for VLM2Vec on the same base 2504.17432 tab. 1. The baselines were re-run by the authors, and the stage-1 code is not released (see the card's Paper vs code).

Think-Then-Embed adds a reasoning step in front of the embedder 2510.05014 §4. A multimodal LLM writes a short reasoning text about the query or the target, and the embedder encodes the original input together with that text. There are three variants: a large teacher reasoner (TTE_t), a small reasoner fine-tuned on the teacher's traces (TTE_s), and one model whose embedding head reads the same pass that writes the reasoning (TTE_u) 2510.05014 §4.2 2510.05014 §4.3 2510.05014 §4.4. Fine-tuning the reasoner matters in the paper's ablation: at 7B the MMEB-V1 score is 73.3 with the fine-tuned reasoner and 68.3 when the backbone writes its own reasoning 2510.05014 tab. 3. On MMEB-V2, the teacher variant reaches 71.5 overall, against 61.2 for the authors' own reproduction of VLM2Vec-V2-7B 2510.05014 tab. 2 2510.05014 app. D. A query is a new item at search time, so the reasoner is in the online path 2510.05014 §4.2, and the paper gives no latency.

Action outputs for robots and vehicles

Two papers fine-tune a multimodal LLM to output a few discrete actions, and a conventional controller carries them out.

ManipLLM takes an image and a prompt and outputs a contact pixel and two gripper direction vectors. Each direction component is one of 100 bins over [-50, 50], so the output is a classification 2312.16217 §3.1.2. On 10 unseen simulated categories it reaches 0.54 initial-movement success against 0.41 for the best baseline 2312.16217 tab. 1. The released code does not include the impedance search or the test-time adaptation the paper describes (see the card's Paper vs code).

DriveMLM outputs one speed decision from four options and one path decision from five, plus an explanation, and a motion planner executes the decision 2312.09245 §3.2. On CARLA Town05 Long, the authors report a driving score of 76.1 against 71.4 for Apollo 2312.09245 tab. 4. The training data is available only on request and no code is released (see the card's Pitfalls).

Slices and tiles for volumes and whole slides

Used by MedGemma 1.5. A 4B model with a frozen MedSigLIP encoder takes a 3D scan as a sequence of axial slices, each rescaled to 896 x 896 pixels. A query holds at most 85 slices, which is 21,760 vision tokens, and longer stacks are sampled at equal spacing along the z-axis [arxiv:2604.05081 §2.3.1]. CT slices are mapped to three colour channels with three Hounsfield windows, while MRI volumes are normalised per volume with no window 2604.05081 §2.3.1. A whole-slide image is cut into non-overlapping 896-pixel patches over the tissue, and at most 126 are kept, chosen at random and put back in their original order 2604.05081 §2.3.2.

The authors report 3D CT accuracy of 61.1 and 3D MRI accuracy of 64.7, against 58.2 and 51.3 for MedGemma 1 4B 2604.05081 tab. 4. These are single runs at temperature 0, and the training sets are mostly internal, so only the public evaluation sets can be rerun 2604.05081 tab. 1 tab. 2. The authors say the model is a research starting point and not for clinical use 2604.05081 §4.

Interleaved features from two vision encoders

Used by Eyes Wide Shut, which also builds the MMVP benchmark. The authors search for image pairs that CLIP ViT-L/14 scores as nearly identical (cosine similarity above 0.95) while DINOv2 ViT-L/14 scores them as clearly different (below 0.6) 2401.06209 §2.1. The 150 pairs give 300 questions, and a pair counts as correct only when both of its questions are right, so random guessing scores 25% 2401.06209 §2.2 2401.06209 §2.3. Human participants average 95.7% [arxiv:2401.06209 §2.3]. GPT-4V scores 38.7% in one run 2401.06209 app. B.3, and most other models are below 25% 2401.06209 §2.3. The best CLIP model on the paper's CLIP-based variant reaches 37.8 2401.06209 tab. 1.

The fix tested on LLaVA is to pass both encoders' tokens to the LLM. In interleaved MoF, each encoder has its own adapter, and the two token sequences alternate in spatial order 2401.06209 §4.3. With LLaVA-1.5 at 224 px, the interleaved model reaches 28.0 on MMVP with 512 visual tokens, against 24.7 for LLaVA-1.5 at 336 px with 576 tokens 2401.06209 tab. 3. At 224 px, the interleaved model gives the LLM 512 visual positions where plain LLaVA gives 256 2401.06209 tab. 3. On wider benchmarks the 224 px interleaved row is lower on some tasks, for example MMBench at 61.6 against 67.7 2401.06209 tab. 7. The released scripts run the interleaved path only through the LLaVA-1 setup, and every shipped script uses 336 px, so the 224 px rows are not reproduced by them (see the card's Paper vs code).

Supervisor VLM that checks gripper changes of a VLA

A vision-language-action policy normally runs its predictions without a check. FPC-VLA adds a Qwen2.5-VL 7B supervisor that is queried only when the gripper state changes by more than 0.5 [arxiv:2509.04018 §3.5] 2509.04018 §4.1. It answers Yes or No, and on No it gives a directional correction for x, y, z and yaw 2509.04018 §3.5. Before the supervisor sees an action, the policy's recent predictions are averaged by a dual-stream fusion 2509.04018 §3.3.

On SIMPLER WidowX the authors report 64.6 average task success, against 51.8 for CogACT, the best baseline in their table 2509.04018 tab. 3. In their four-task ablation, removing the supervisor lowers task success from 82.1 to 74.4 2509.04018 tab. 7. The supervisor is trained mostly on synthetic pairs, 100k per robot against 10k annotated entries [arxiv:2509.04018 §4.1]. The card records no official code (see the card's Paper vs code).

Slow VLM planner with a fast diffusion driver

DualVLN splits vision-and-language navigation into two systems. System 2 is a Qwen2.5-VL 7B planner that outputs a pixel goal, or a view-adjustment turn when no future waypoint is visible 2512.08186 §3.1. System 1 is a diffusion transformer that outputs 32 waypoints. It is conditioned on four latent tokens from System 2 and on a 32-token summary of two RGB frames 2512.08186 §3.2 2512.08186 §3.3. The two systems run asynchronously, so the driver keeps updating from newer frames between planner calls 2512.08186 §3.2.

On R2R-CE val-unseen with RGB only, the authors report 64.3 success rate against 56.9 for StreamVLN 2512.08186 tab. 1. The timing figures are 0.7 s per planner output and 0.03 s per driver call on one RTX 4090, both author measurements 2512.08186 §5.2. The released code differs from the paper in several details, such as a scale factor on the trajectory and the sampler settings, so a reimplementation should follow the card's Paper vs code.

Frozen VLM hint for a small imitation driver

Used by Zero-shot LLM driving. A ViT-B/16 policy is trained by imitation on one cone in a hallway. It has one steering and throttle head for each of three route classes, LEFT, MIDDLE and RIGHT [arxiv:2411.14256 §2.1] 2411.14256 app. B.1. A frozen multimodal LLM is not fine-tuned. It sees the same camera frame with a five-step chain-of-thought prompt and answers with one of the three words 2411.14256 §2.1 [arxiv:2411.14256 tab. 1]. The policy acts on every frame and keeps the last word it received while a new answer is pending 2411.14256 §2.2.

The LLM is slow. Its answers take between 5.9 and 7.8 seconds on the authors' hardware 2411.14256 tab. 2, so the car would stall without the cache. On a harder zigzag course with cones and bins, the policy alone succeeds in 40% of real runs, and with an LLM hint the success rate is 63% to 83%, depending on the model 2411.14256 tab. 3. Those are 30 runs per configuration on one course 2411.14256 §3.3. The paper does not say which instruction the policy uses before the first answer 2411.14256 §2.2.

Post-training with verifiable rewards

Two papers train a multimodal model with reinforcement learning on rules that can be checked automatically, rather than on copied answers. Both use GRPO on Qwen-based models and both report their own numbers.

Visual-RFT fine-tunes Qwen2-VL with rewards for box overlap and confidence, exact class match for classification, and a format check 2503.01785 §3.2. On one-shot fine-grained classification over four datasets, the authors report an average of 80.3 for Visual-RFT, 51.7 for SFT and 56.0 for the 2B base model 2503.01785 tab. 2. On 2-shot COCO detection with the 2B model, the mAP goes from 19.6 to 41.5 [arxiv:2503.01785 tab. 3]. The paper gives no learning rate and no IoU threshold, and the code sets the threshold to 0.5 with a strict comparison (see the card's Paper vs code).

VideoChat-R1 fine-tunes Qwen2.5-VL-7B on a joint set of temporal grounding, object tracking and grounded QA, 18,031 samples in total 2504.06958 §3.3. Its rewards are an IoU for time ranges and boxes, an answer-accuracy check, and a format check 2504.06958 §3.2. The authors report Charades-STA mIoU rising from 29.0 to 60.8 and GoT-10k overlap from 12.6 to 43.8 over the base model 2504.06958 tab. 1. In their ablation, three epochs of SFT lower ActivityNet mIoU from 20.6 to 17.3 [arxiv:2504.06958 tab. 3]. The released grounding and tracking code does not include the caption-recall reward or the clue-based inference the paper describes (see the card's Paper vs code).

Continual instruction tuning

Training a model on tasks in sequence makes it forget earlier ones. SAME studies this for LLaVA-v1.5-7B. The vision encoder and the projector are frozen, and only LoRA experts in the language model's feed-forward layers, with a router, are trained 2602.01990 §3 2602.01990 §5.1. The method stores no samples from earlier tasks 2602.01990 §1. It has three parts 2602.01990 §4: a router update limited to the input directions that matter for the current task, expert updates scaled by the earlier tasks' input covariance, and a per-task freeze of experts that the current task barely uses 2602.01990 §4.1 2602.01990 §4.2 2602.01990 §4.3.

The author-reported averages are 66.82 on CoIN against 50.58 for MoELoRA 2602.01990 tab. 2, 67.12 on UCIT against 52.06 for MoE-LoRA 2602.01990 tab. 3, and 46.53 on the authors' ten-task TriGap benchmark against 44.45 2602.01990 tab. 1. On CoIN, the ablation rises from 50.58 to 61.32 with the router part, to 65.89 with the expert part, and to 66.82 with freezing 2602.01990 tab. 4. The paper reports single runs with no error bars 2602.01990 §5.1. The CoIN baselines are taken from their papers, while the TriGap baselines were re-run 2602.01990 tab. 2 2602.01990 tab. 1. The hyperparameters were picked on a 20% sample of UCIT's training data and then reused for every benchmark [arxiv:2602.01990 app. F].

The released code routes once per forward pass rather than per token, and its per-task reset is never called (see the card's Paper vs code).

Reasoning before the answer, with a picture

Two papers test whether a model should write its reasoning before its answer when the question includes a picture. Both use ScienceQA, and they disagree on the order.

The ScienceQA paper (Lu et al.) builds 21,208 multiple-choice science questions, many with a lecture and an explanation, and prompts models to write the answer first, then the lecture and explanation 2209.09513 §3 §4.2. Two-shot GPT-3 in that format scores 75.17 on the test split, 1.20 points above two-shot GPT-3 without the reasoning 2209.09513 tab. 3. On a 1,000-item test, answer-first output scores 73.6 against 60.6 when the lecture comes first 2209.09513 tab. 7. With gold explanations in the prompt the score reaches 94%, which shows that explanation quality matters but is not a method 2209.09513 tab. 6.

Multimodal-CoT splits the job between two models. Each is a T5 initialised from FLAN-Alpaca. The first writes a rationale from the text and frozen ViT-L/32 features, and the second answers from the text plus that rationale 2302.00923 §4.1. The picture enters through single-head attention and a sigmoid gate 2302.00923 §4.2. In a text-only two-stage pilot, the authors read 56% of 50 sampled errors as rationales that mislead the answer step 2302.00923 §3.2, and they say the picture features correct 60.7% of those hallucinated-rationale mistakes 2302.00923 §3.3. In a 200M text-only pilot, both rationale-first (69.32) and answer-first (69.68) lower accuracy against no reasoning (81.63) 2302.00923 tab. 2. With the picture features, the Base model (223M) reaches 85.31 and the Large model (738M) 90.45 on the ScienceQA test split 2302.00923 tab. 4. Generated rationales from InstructBLIP and ChatGPT, used in place of annotated ones, give 87.76 on the Large model against 90.45 2302.00923 tab. 7.

Inference cost: dropping image tokens

Image tokens are a large share of the input. FastV ranks the image tokens by attention after an early layer, keeps the top share and drops the rest for the later layers. It needs no training 2403.06764 §4.1. The headline setting, K=2 and R=50%, keeps the four-benchmark average of LLaVA-1.5-13B at 73.6 with about 55% of the theoretical FLOPs 2403.06764 tab. 1. The one real timing measurement is on A-OKVQA with one-token answers, on a single A40. The 13B model with FastV takes 0.341 s per example against 0.344 s for the unmodified 7B model 2403.06764 tab. 4.

The FLOPs figures come from a formula, not from timing 2403.06764 §4.2. The released code ranks tokens by a different score from the one the paper states, and its demo uses different settings from the headline (see the card's Paper vs code).

Streaming video adds a memory problem. HERMES splits the KV cache of a streaming video model into three roles by depth: shallow layers favour recent tokens, deep layers keep one anchor per frame, and middle layers blend the two. It keeps a fixed token budget per layer and merges the dropped tokens of deep layers into one summary token 2601.14724 §2 2601.14724 §3.2. It needs no training. With Qwen2.5-VL-7B at a 4K budget, the authors report 79.44 on StreamingBench real-time against 73.31 for the base 2601.14724 tab. 1. The offline gains are mixed: the VideoMME average for the same model falls from 64.52 to 60.63 2601.14724 tab. 4. The latency ratios in the paper come from a comparison chart with no values in the text 2601.14724 §4.3.

Object hallucination at decoding time

Object hallucination means naming objects that are not in the image 2311.16922 §1. Visual contrastive decoding changes only the sampler. At each step it computes the next-token logits twice, once for the image and once for a copy with Gaussian noise added 2311.16922 §3.2 2311.16922 §3.3.1. It then contrasts them as (1+α)·logit(image) − α·logit(noised image), with α=1 2311.16922 eq. 3 2311.16922 §4.1. Only tokens whose probability under the clean image is at least β=0.1 times the top probability are kept 2311.16922 §3.3.2 2311.16922 eq. 4.

The authors report up to +5.8 accuracy and +7.4 F1 on POPE across LLaVA-1.5, InstructBLIP and Qwen-VL 2311.16922 §4.2 2311.16922 tab. 1. For LLaVA-1.5 on MSCOCO random, accuracy goes from 83.29 to 87.73 2311.16922 tab. 1. On the MME hallucination subset, LLaVA-1.5 totals 565.33 against 604.66 2311.16922 tab. 2. On LLaVA-Bench, with answers judged by GPT-4V, Qwen-VL's accuracy score goes from 4.76 to 6.69 2311.16922 tab. 3.

The method needs a second forward pass per generated token. The authors call the overhead minimal, but report no timing or memory numbers 2311.16922 §1. The paper's β sweep on MME scores β=0.5 higher on the hallucination subset, 635.00 against 591.67 at the default 0.1, and the paper does not explain the default 2311.16922 app. B. Its noise is described as a multi-step process with a fixed γ, while the released code uses a different schedule, and the paper does not say which one produced its numbers (see the card's Paper vs code). The tests cover three models at 7B scale, plus 13B variants of two of them 2311.16922 app. B.

Multi-agent workflows

SurgRAW puts several prompted models around a vision-language model, with no training. An orchestrator routes each question to a visual or a knowledge path. Visual answers pass through a panel that checks an instrument-action table and rates the answers, with up to three refinement runs 2503.10265 §3.2 2503.10265 §3.3. Knowledge questions receive retrieved medical text 2503.10265 §3.3. The authors report 61.73% overall for SurgRAW-GPT4o against 31.44% for GPT-4o alone, on their own five-task benchmark of robotic surgery frames 2503.10265 tab. 3. The benchmark is built from GPT-4o questions and reviewed by two clinicians, with no error rate reported 2503.10265 §3.1.2. The repository has no evaluation script, so the accuracy numbers cannot be checked from it (see the card's Paper vs code).

Safety of multimodal models

A picture can carry a request that the text alone would not. MM-SafetyBench builds 1,680 harmful questions in 13 policy scenarios, each paired with three generated images: a Stable Diffusion picture of the key phrase, a typography picture of the phrase, and the two stacked 2311.17600 tab. 1 2311.17600 §3.2. A GPT-4 judge labels each answer safe or unsafe 2311.17600 §3.3. For LLaVA-1.5-7B, the average attack success rate rises from 41.01 with text only to 71.52 with typography, and to 72.14 with the stacked image 2311.17600 tab. 3. A fixed safety sentence placed before the question lowers the stacked-image average to 17.08 2311.17600 tab. 4.

The paper's text gives a different figure for the tiny subset, 15.68, which appears in no table 2311.17600 §3.4. Use the table values. The data is released under non-commercial terms, so reuse the method, not the images or questions (see the card's Pitfalls).

Small pixel changes are a second risk. Schlarmann and Hein attack OpenFlamingo 9B white-box with APGD, bounding the change to 1/255 or 4/255 in the infinity norm 2308.10741 §4 2308.10741 §5. At 4/255, the targeted attack reaches 100% success for one chosen caption in both zero-shot and 4-shot settings, and 83.80% zero-shot for the second [arxiv:2308.10741 tab. 2]. Untargeted COCO CIDEr falls from 84.01 to 1.69 zero-shot [arxiv:2308.10741 tab. 3]. The study covers one model and no transfer to other systems.

Omni-SafetyBench renders 972 harmful seed prompts into 24 text, image, video and audio combinations, giving 23,328 cases [arxiv:2508.07173 §2.2] 2508.07173 tab. 2. Each response is judged for comprehension, harm and refusal, and only responses the model understood enter the rates 2508.07173 §2.3.1. Of eleven models, the authors report that three exceed 0.6 on both the average Safety-score and the cross-modal consistency score 2508.07173 §3.2 2508.07173 tab. 8. The paper's labels come from a majority of five LLM judges, but the public script uses one judge and does not compute the Safety-score or the consistency score (see the card's Paper vs code).

A closed omni-model is audited in a different way. The GPT-4o system card 2410.21276 §1 covers speech, image and video inputs, and its safety work centres on voice: the voice-output classifier reaches precision 0.96 and recall 1.0 on English 2410.21276 tab. 1, and refusals for speaker identification rise from 0.83 to 0.98 in the authors' table [arxiv:2410.21276 tab. 2]. The audio evaluations are text benchmarks converted to speech with a TTS system, so the scores also reflect the TTS system 2410.21276 §3.2. The card gives no architecture, learning rate or weights, so it documents the evaluation design rather than the method 2410.21276 §2.

Knowledge questions about a pictured entity

InfoSeek asks questions whose answers are not visible in the picture, such as when a building was constructed 2302.11713 §1. It has 8.9K human-written questions and about 1.35M generated from Wikidata 2302.11713 §3.1 2302.11713 §3.2. Zero-shot, the pre-trained models score low. After fine-tuning, PaLI-X reaches 22.1 overall on the Wikidata split and 10.8 on the human split, while an entity recognizer followed by a Wikipedia reader reaches 18.2 on the human split 2302.11713 tab. 4 2302.11713 tab. 5. The metric in the paper differs from the released evaluation code, so the two may not reproduce each other (see the card's Paper vs code).

Reading text in pictures

Reading the words in a photo or a document is a skill separate from describing a scene. OCRBench tests it with 1000 question-answer pairs in five jobs: text recognition (300 items), scene-text VQA (200), document and chart VQA (200), key-information extraction (200) and handwritten maths (100) 2305.07895 §3.3 2305.07895 tab. 5. A reply scores 1 if a ground-truth string appears in it, and 0 otherwise. The final score is the sum over the 1000 items, so its maximum is 1000 2305.07895 §3.1 2305.07895 tab. 3. The released scorer also ignores letter case, which the paper does not state (see the card's Paper vs code).

The authors' zero-shot runs give Gemini 659 and GPT4V 645 out of 1000, with the open Monkey model at 514 2305.07895 tab. 3. GPT4V refused 84 images on safety grounds, and a refusal scores 0 2305.07895 §3.3. Substring matching lets a long reply pass if it contains the answer, and it does not normalise dates, so "2 February 2018" fails for "02/02/2018" 2305.07895 §3.3 (see the card's Pitfalls).

Long-video evaluation

Video models often pass tests that a few seconds of footage can answer. EgoSchema is a five-option, multiple-choice benchmark of about 5,000 questions on three-minute first-person clips 2308.09126 §3.1 2308.09126 §4.2. The authors measure how much of a clip a human must watch to verify an answer, with a median of about 100 seconds 2308.09126 §4.1. Published video-language models score between 19.6% and 32.1% against 20% chance, while people score about 75% to 76% 2308.09126 §4.2. The answer key is mostly withheld, and the video comes from Ego4D (see the card's Pitfalls).

Video-ChatGPT uses a different measure, a GPT-3.5 judge on a five-aspect video chat benchmark. It gives Video-ChatGPT a correctness score of 2.40 on a 1-to-5 scale, the top score among the compared models 2306.05424 tab. 1. The judge is a closed model, so the scores move with the judge [arxiv:2306.05424 tab. 4]. Scores from the two benchmarks are not comparable.

MVBench takes a different route: it turns 9 families of static image questions into 20 temporal tasks, each with 200 multiple-choice items drawn from 11 public video datasets 2311.17005 §3.1 §3.2. The answer is scored as the option letter that follows a fixed prefix, so the scoring does not need a judge model 2311.17005 §3.3. The authors report 51.1 average accuracy for VideoChat2, a Vicuna-7B model trained in three stages, against 35.5 for VideoChat 2311.17005 tab. 2. Frame count and resolution differ across the rows of that table, and GPT-4V was run at 512 x 512 with 16 frames, so the GPT-4V comparison is not like for like 2311.17005 tab. 2. A variant fed a blank video and no LoRA weights scores 34.7, close to VideoChat's 35.5 2311.17005 §5.1.

Spatial memory and layout from video

Video models are usually tested on what happens in a clip. VSI-Bench tests whether they can recall the layout of a room they have walked through. It has over 5,000 question-answer pairs from 288 indoor videos, taken from the validation sets of ScanNet, ScanNet++ and ARKitScenes 2412.14171 §3.1 2412.14171 app. B.1. The eight tasks cover object count, absolute distance, object size, room size, relative distance, relative direction, route plan and appearance order 2412.14171 §3.2. Multiple-choice answers are scored by accuracy, and numerical answers by mean relative accuracy over ten thresholds 2412.14171 §4.1 2412.14171 eq. 1.

The authors report that humans reach 79.2 on a 400-question subset, and the best model, Gemini-1.5 Pro, 48.8 on the same subset 2412.14171 tab. 1. Seven of the twelve open-source models score below the frequency baseline on average 2412.14171 §4.2 2412.14171 tab. 1. The models see different input: Gemini gets the whole video, and open-source models get 8 to 32 uniformly sampled frames 2412.14171 app. C.3. Humans score 94 to 100 on the configurational and appearance-order tasks, and the gap to the best model is narrower on the three measurement tasks 2412.14171 §4.2.

Written reasoning does not help on this benchmark. Zero-shot chain-of-thought, self-consistency and tree-of-thoughts all lower the average against no prompting. The authors give about 4% for the first and third and 1.1% for self-consistency, and those drops appear only in a figure 2412.14171 §5.2 2412.14171 fig. 8. Self-explanations of 163 wrong answers put about seven in ten errors under spatial reasoning 2412.14171 §5.1.

An explicit map helps on distance questions. Gemini-1.5 Pro is first asked to place each object's centre on a 10 x 10 grid, and then to answer the relative-distance question from that map. Accuracy rises from 46.0 to 56.0 2412.14171 tab. 3. A 20 x 20 grid gives 54.0 with the model's own map. Maps built from ground-truth annotations give 66.0, or 78.0 at 20 x 20, which the authors treat as an upper bound and not a method 2412.14171 tab. 3. The second-stage prompt is not stated 2412.14171 §6.2. The model places adjacent objects correctly within one grid unit in 64% of cases, and accuracy falls with distance 2412.14171 app. B.5.

Which input the answer relies on

Sensory PID splits what video and audio each contribute to an answer, with the text instruction held fixed, into a video-only part, an audio-only part, a shared part and a synergy part 2606.00959 §2.1 2606.00959 eq. 1. It needs no training. Each stream is replaced with noise in turn, and the answer probabilities are fitted with the BATCH estimator 2606.00959 §2.2. On MUSIC-AVQA, the visual-only term is larger than the audio-only term in most subsets. On the audio-focused subset, the audio term is larger for VITA-1.5 and Qwen2.5-Omni 3B, and nearly equal to the visual term for Qwen2.5-Omni 7B 2606.00959 tab. 1. The study covers three omni-modal models and one benchmark.

The same decomposition gives sample weights for vision-language fine-tuning. Over three seeds, Qwen2.5-VL-7B reaches 64.3 on MMStar with the PID weights against 62.0 with uniform weights, a gap of 2.3 points with a 95% interval of [1.4, 3.2] 2606.00959 tab. 2 2606.00959 §5.1. On Gemma3-12B the MMStar score falls from 56.2 to 54.3 2606.00959 tab. 3. The authors do not claim that the synergy shift causes the gain 2606.00959 §5.2.

Training data

• Image-text pairs for pretraining. LAION-400M is about 400 million pairs from Common Crawl alt-text, kept when the CLIP image-text cosine similarity is at least 0.3 2111.02114 §2.1.1. The paper lists 413M unique samples 2111.02114 tab. 1. It reports no benchmark result for the dataset itself 2111.02114 §3. The dataset's license is not stated, and the CLIP filter is not in the linked code, so the filter cannot be rebuilt from it 2111.02114 §2.1.1.
• Video instruction data. Video-ChatGPT's 100K pairs are 30% human-written and 70% semi-automatic, the latter made with BLIP-2, GRiT, Tag2Text and GPT-3.5 2306.05424 §4.2 2306.05424 §5.3. The paper says the data will be released under MIT terms, which the card has not verified 2306.05424 §9.
• Pretraining corpus for a detail-heavy encoder-decoder. Florence-2's FLD-5B has 126M images and 5.4B annotations 2311.06242 §4. Its labels come from specialist models, including cloud services, in three phases: initial annotation, filtering and iterative refinement 2311.06242 §4.2. The paper does not state the license of FLD-5B itself, and the labelling models' licenses are not stated either 2311.06242 §4.1.
• Medical multi-task benchmark. MultiMedBench, used by Med-PaLM M, has 14 tasks over 12 datasets and more than one million samples 2307.14334 §3. Report generation on MIMIC-CXR is 59.90% of the fine-tuning mix 2307.14334 tab. A.1. The terms of the clinical datasets were not checked, so check each one before reuse.
• Multi-target segmentation data. PixelLM's MUSE has 246k question-answer pairs over about 910k LVIS instances 2312.02228 §4.1. Its labels are made with GPT-4V and checked with GPT-4V and humans 2312.02228 §4.2.
• Instruction samples from captions and boxes. LLaVA's 158K samples are 58K conversations, 23K detailed descriptions and 77K reasoning questions, written by GPT-4 from COCO captions and boxes, without the image [arxiv:2304.08485 §3]. The project page lists the data as CC BY-NC 4.0, so do not use it for commercial training without a separate licence check [arxiv:2304.08485 Pitfalls].
• Referential dialogue. Shikra's generated set has 5,922 question-and-answer pairs, written by GPT-4 from Flickr30K captions and boxes rather than from images 2306.15195 §5.3.2.
• Science multiple choice. ScienceQA has 21,208 questions, split 60:20:20 into train, validation and test 2209.09513 §3 app. A.3. Its licence is CC BY-NC-SA 4.0, for non-commercial research only, so the dataset cannot be reused commercially 2209.09513 app. A.1.
• Clinical sets. Most of MedGemma 1.5's training and several evaluation sets are internal, so only the public sets in its table can be reproduced 2604.05081 tab. 1 tab. 2.
• Speech-text instruction pairs. SpeechInstruct's cross-modal part turns about 9 million English ASR pairs into recognition and synthesis tasks, with instruction wordings written by GPT-4 2305.11000 §3.1. The chain-of-modality part converts 37,969 text instruction responses to speech with a generator trained on LibriSpeech 2305.11000 §3.2. The card records no licence for the dataset, so check the source corpora before reuse.
• Synthetic route instructions from video. InstruGen takes routes through 3,879 YouTube house tours, labels rooms with CLIP and estimates the action between rooms with an inverse model [arxiv:2411.11394 §3.1]. A multimodal model writes the instruction. A consistency check then extracts room-and-action pairs from the text and regenerates the instruction when they do not match the route 2411.11394 §3.2. The paper does not state how pairs are compared, how many retries are allowed, or how many instructions are generated 2411.11394 §3.2 2411.11394 §4.1.
• Question-and-answer pairs for robots in crowds. Social-LLaVA annotates about 2K robot-human scenes from the SCAND navigation data, with over 40K questions and answers on perception, prediction, reasoning, action and explanation 2501.09024 §III-B 2501.09024 §I. The paper does not say that the set is released, and it does not state the license of SCAND 2501.09024 §III-B.
• Spatial questions generated from 3D scans. RoboSpatial builds questions from indoor and tabletop scans, with boxes and images from EmbodiedScan 2411.16537 §4.1. Three relation types are generated: free points in a direction from an anchor object, whether a virtual box of the target's size fits, and whether one object lies left, right, above, below, in front of or behind another. Each relation is read in the camera, world or object frame 2411.16537 §3.1 2411.16537 §3.2. Training uses 900,000 relations, and LLaVA-NeXT's average on the authors' validation split rises from 38.1 at 100K relations to 72.4 at the full 3M [arxiv:2411.16537 app. B.1] 2411.16537 tab. 9. The source scan sets carry their own terms, which the card did not check, and the official code is licensed for research and evaluation only (see the card's Pitfalls and the card's Paper vs code).

When to use which

Implementation sketch

The shared core is: encode the image or video, map the features into the decoder's input space, prepend them to the text, and run the decoder. The variants differ in the bridge, in which parts train, and in what the decoder writes. The Gemini report does not describe its encoder or bridge, so the sketch follows InstructBLIP. Florence-2 differs in one place: its visual vectors go into the encoder of an encoder-decoder, and a decoder generates the answer 2311.06242 §3. Video-ChatGPT replaces the Q-Former with pooling and one linear layer 2306.05424 §3.1. PandaGPT replaces the encoder with a shared embedding space and adds a single vector per input 2305.16355 §3. Med-PaLM M updates every weight, so there is no frozen part to skip 2307.14334 §4.2. ViP-LLaVA changes the input rather than the bridge: it draws the marks onto the pixels before encoding 2312.00784 §3.1. UniME reads the last token's hidden state as the embedding 2504.17432 §3.2. LLaVA and MiniGPT-4 keep the sketch's linear map and drop the query bridge 2304.08485 §4.1 2304.10592 §3. OFA and PaLI have no separate bridge: OFA writes image codes and box tokens into one vocabulary 2202.03052 §3.1, and PaLI feeds ViT patch features to an mT5 encoder-decoder, where the interface is not fully specified 2209.06794 §3.1. ClipCap keeps a bridge but makes it a small mapper that writes prefix vectors from one CLIP embedding 2111.09734 §3.1. SpeechGPT has no bridge at all: speech units are vocabulary tokens 2305.11000 §4.2. SpatialRGPT adds one bridge input per region: the mask-averaged features of that region, placed where the prompt names it, plus an optional depth vector from a second connector 2406.01584 §3.3.

# Shared core: image features enter as input vectors  [arxiv:2305.06500 §2.3]
# Input:  image, text tokens (the instruction or prompt)
# Output: logits over the next text token

def multimodal_forward(image, text_tokens):
    # Frozen image encoder, ViT-g/14 in InstructBLIP  [arxiv:2305.06500 §2.6]
    image_feats = image_encoder(image)                 # (N, d_v)

    # Bridge: learned queries read the image and the instruction
    # [arxiv:2305.06500 §2.3]. Trainable in InstructBLIP. Gemini's bridge
    # is not described  [arxiv:2312.11805 §2]
    query_out = bridge(queries, text_tokens, image_feats)   # (K, d_q)

    # Linear map into the decoder's input space  [arxiv:2305.06500 §2.3]
    visual_vecs = linear_projection(query_out)         # (K, d_llm)

    # Prepend the visual vectors to the embedded text  [arxiv:2305.06500 §2.3]
    embeds = concat(visual_vecs, embed(text_tokens))

    # Frozen decoder predicts the next token  [arxiv:2305.06500 §2.2]
    return decoder(embeds)

Contradictions

• FlanT5-XL size, within InstructBLIP. The method section gives FlanT5-XL as 3B parameters 2305.06500 §2.6. The zero-shot results section calls the same model 4B 2305.06500 §3.1. The paper does not resolve this.
• TextVQA across papers. InstructBLIP's FlanT5-XL reaches 46.6 on TextVQA as a held-out, zero-shot dataset 2305.06500 tab. 1. Gemini Ultra reaches 82.3% on TextVQA (val), 0-shot, pixel input only, using the post-trained API model 2312.11805 tab. 7. These are different models, splits and settings, so the numbers are not comparable. Neither report reconciles them.
• Parameter counts, within Florence-2. The text gives 232M for base and 771M for large 2311.06242 §6.1. The layer breakdown in app. C sums to 230M and 770M 2311.06242 app. C. The paper does not explain the gap, so quote the table or the text, and say which.
• Frozen encoder, within Florence-2. The text says freezing the encoder does not affect image-level tasks 2311.06242 §6.5. The ablation shows the same encoder at 6.9 COCO AP against 19.7 by default, while caption CIDEr rises slightly 2311.06242 tab. 12. The claim holds for captions only.
• Count of LAION-400M, within the dataset paper. The abstract and introduction say 400 million pairs 2111.02114 §1. Table 1 lists 413M unique samples 2111.02114 tab. 1. The paper does not say which count is released.
• Clinical errors per report, within Med-PaLM M. The introduction and the radiologist results give 0.25 clinically significant errors per report for the best model 2307.14334 §1 2307.14334 §6.3.2. The appendix table gives 0.23 for significant clinical errors and 0.25 for total clinical errors, both at 84B 2307.14334 tab. A.8. The 0.25 in the text matches the total-clinical-errors row, not the significant-errors row. The appendix text also calls the 12B model the best performer, on total errors, while the table shows the 84B model with the fewest clinical errors 2307.14334 app. A.4 2307.14334 tab. A.8.
• Architecture naming, within Med-PaLM M. The introduction calls the model sequence-to-sequence 2307.14334 §1. The PaLM base is a decoder-only Transformer 2307.14334 §4.1. The two descriptions cannot both be right, and the paper does not reconcile them.
• Safety-prefix figure, within MM-SafetyBench. The text gives 77.33 to 15.68 for LLaVA-1.5-7B on the tiny set 2311.17600 §4.5. The table value 77.33 belongs to the Vicuna-7B row of another table, and 15.68 appears in no table 2311.17600 tab. 5. The default-set table gives 72.14 to 17.08 2311.17600 tab. 4.
• Token-pruning claim, within FastV. The abstract and the results text say the headline setting costs no performance. The MME total drops from 1143.5 to 1129.8 for InstructBLIP at the same setting, and the paper says it needs K=5 to close the gap 2403.06764 tab. 2 2403.06764 §5.3.
• Token-fusion gain, within PixelLM. The text says token fusion gives up to 2.4% cIoU on MUSE val 2312.02228 §5.5. The ablation table gives a gain of 2.1 2312.02228 tab. 4.
• Headline gap, within UniME. The text gives a 10.3% gain over VLM2Vec with LLaVA-1.6 2504.17432 §4.3. Table 1 gives 66.6 against 63.3, a gap of 3.3 points 2504.17432 tab. 1. The paper does not reconcile the two.
• Visual7W, within ViP-LLaVA. The main table gives 87.91 for the 13B model 2312.00784 §5.2. The GPT-4V-data appendix table gives 88.28 for the 13B model 2312.00784 app.. The paper does not explain the difference.
• LLaVA accuracy, within DriveMLM. The text gives 40.97% [arxiv:2312.09245 §4.3], and the table gives 22.92% for the same model 2312.09245 tab. 3.
• MM-CoT Large on ScienceQA, across papers. Multimodal-CoT reports 90.45 for its Large model 2302.00923 tab. 4. LLaVA's table lists "MM-CoT Large" at 91.68, marked as a literature value 2304.08485 tab. 7. The two papers do not explain the gap.
• Order of reasoning and answer on ScienceQA. The ScienceQA paper finds the answer-first format better: 73.6 against 60.6 when the lecture comes first 2209.09513 tab. 7. LLaVA's ablation finds the reverse on ScienceQA: answer-first gives 89.77 against 90.92 for reasons first [arxiv:2304.08485 tab. 8]. The setups differ in model, data and fine-tuning, and neither paper reconciles them. Multimodal-CoT writes the rationale first [arxiv:2302.00923 §4.1].
• Multimodal-CoT model size, within the paper. The text gives 200M and 700M parameters, and Table 4 gives 223M and 738M 2302.00923 §5.2 tab. 4. The paper does not explain the difference.
• Stage 2 length, within SpeechGPT. The text gives 2,100 steps for cross-modal instruction tuning 2305.11000 §5.1, and the appendix table gives 4,000 2305.11000 app. D. The paper does not reconcile them.
• FloodNetVQA on TriGap, within SAME. The text describes the gain on FloodNetVQA as clear 2602.01990 §5.2, but the table gives MoE-LoRA 90.41 against 81.09 for SAME there 2602.01990 tab. 1. The paper does not reconcile the two.
• Gemini on document reading, across papers. The Gemini report gives 90.9% on DocVQA for the post-trained Ultra model 2312.11805 tab. 7. OCRBench scores its Gemini entry, which is gemini-pro-vision, at 128 of 200 on its document VQA items 2305.07895 tab. 3 2305.07895 tab. 6. These are different models, data and scoring rules, so the numbers are not comparable. Neither paper reconciles them.
• Stage-2 trainable parts, within mPLUG-Owl. The abstract and §1 describe LoRA on the LLM together with the abstractor 2304.14178 §1. §3.2 freezes the abstractor in stage 2 and trains LoRA on the LLM alone 2304.14178 §3.2.
• Stage-1 hyperparameters, within mPLUG-Owl. The text gives learning rate 0.0001, 2,000 warmup steps, weight decay 0.1 and a batch of 2.1M tokens 2304.14178 §4.1. The appendix table gives 2e-4, 375 steps, 0.01 and 4,096 image-caption pairs 2304.14178 app. A.
• Trainable size, within Otter. The text gives 1.4B trainable parameters 2305.03726 §3.1. Table 2 gives 1.3B for Otter-Image [arxiv:2305.03726 tab. 2].
• Video frame count, within Otter. The text samples 64 frames for training 2305.03726 §6.1. The hyperparameter table gives 8 [arxiv:2305.03726 tab. 11].
• Location-bin ranges, within VisionLLM. The main text gives 1,025 location tokens covering -512 to 512 2305.11175 §3.4. The ablation text describes 257 bins as -128 to 128 and 2,049 bins as -1,024 to 1,024 2305.11175 §4.4.
• Hardware, within MotionGPT. The experiments section says 8 V100 GPUs 2306.14795 §4.1. The appendix says 64 V100 GPUs 2306.14795 app. H.
• Headline claim against the table, within MotionGPT. The abstract calls the results state of the art 2306.14795 §1. The text-to-motion table gives T2M-GPT an FID of 0.116 against 0.232 for MotionGPT 2306.14795 tab. 3, and §4.2 says "competitive".
• Video-first input, within VSI-Bench. The text says video-first input is 2.5% worse 2412.14171 app. D. The table puts video-first at 48.8 and question-first at 46.3 2412.14171 app. D, which reverses the direction.
• Chain-of-thought gain on VideoMME, within VSI-Bench. The text gives a 1.6% improvement 2412.14171 §5.2. The table gives 77.2 without and 79.8 with chain-of-thought, a 2.6-point gain 2412.14171 tab. 2.
• Self-consistency temperature, within VSI-Bench. The main text gives 1.0 2412.14171 §5.2. The appendix gives 0.7 2412.14171 app. B.3.
• LRBEN zero-shot, within GeoChat. Table 6 calls GeoChat "not specifically finetuned on target dataset" 2311.15826 tab. 6. Table 1 lists 56k RSVQA-LRBEN training pairs 2311.15826 tab. 1. The paper does not reconcile the two.
• Best open model, within Think-Then-Embed. The text calls the fine-tuned reasoner the best open model at 2B and 7B 2510.05014 §6.1. Table 2 lists RzenEmbed-v1-7B at 68.9 against 68.6 for TTE_s-7B [arxiv:2510.05014 tab. 2]. The paper does not define "open-source".
• Role of the LLM in the driving hint, within Zero-shot LLM driving. The main text says the language model gives only a high-level instruction and the policy outputs the steering and throttle 2411.14256 §1 2411.14256 §2.1 2411.14256 §4. The appendix says that, with images and instructions, the LLM provides the steering and throttle values and the probability head is dropped 2411.14256 app. B.4. The paper does not say which setup produced the test results.
• Headline MedQA gain, within Med-Gemini. The abstract and introduction give 4.6 points over Med-PaLM 2 2404.18416 §1. The results section gives 4.5 points 2404.18416 §4.1. The paper does not reconcile them.
• Surgical view accuracy, within Med-Gemini. The main table gives 55.2 for the model 2404.18416 tab. 3. The appendix gives 50.0 [arxiv:2404.18416 tab. A1], and the text's 21% figure matches the appendix value.
• Classification claim, within SpatialRGPT. The text says the model beats the region-aware baselines on classification 2406.01584 §4.2. In the classification table, RegionGPT-7B is higher than SpatialRGPT-7B on both mAP and accuracy, and only the VILA-based variants are higher 2406.01584 tab. 3.
• BLINK gain, within SpatialRGPT. The text says the gain over GPT-4V-Turbo on BLINK is more than 20% 2406.01584 §4.2. The 8B model gives 87.9 against 66.9, which fits that claim. The 7B model gives 82.3, a 15.4-point gap 2406.01584 tab. 4.
• Commonsense claim, within RoboSpatial. The text says trained models keep or slightly improve performance on every general benchmark 2411.16537 app. D.3. The MME cognition score falls from 305.4 to 293.2 2411.16537 tab. 10.

Papers

• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs: an open set of about 400M CLIP-filtered image-text pairs with embeddings and search indexes, without a model benchmark.
• Unified-IO: A unified model for vision, language, and multi-modal tasks: one T5-style encoder-decoder that writes images, boxes, masks and depth as discrete tokens, trained on 95 datasets.
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning: instruction-tunes a BLIP-2 model with a frozen image encoder and LLM, training only an instruction-aware Q-Former.
• PandaGPT: One Model To Instruction-Follow Them All: joins a frozen ImageBind encoder to a frozen Vicuna-13B through one linear projection and LoRA, for six input types, with example outputs only.
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models: a frozen CLIP encoder and frozen Vicuna-7B joined by one trainable linear layer over pooled frame features, tuned on 100K video-instruction pairs.
• Towards Generalist Biomedical AI: Med-PaLM M, a PaLM-E model fully fine-tuned on MultiMedBench with text prompts, with a radiology report and classification evaluation.
• EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding: a five-option benchmark on three-minute Ego4D clips, with a temporal-certificate measure of how much video each question needs.
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks: a prompt-driven encoder-decoder that writes vision answers as text, pretrained on FLD-5B from a model-driven data engine.
• MM-SafetyBench: 1,680 harmful questions rendered as images in three ways, showing that pictures can bypass text-level refusals.
• PixelLM: Pixel Reasoning with Large Multimodal Model: an LLM that writes codebook tokens for each target, with a light pixel decoder that turns them into masks, trained on the MUSE dataset.
• Gemini: A Family of Highly Capable Multimodal Models: a family of decoders trained natively on text, image, audio and video, with a benchmark-heavy report and few reproducible details.
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context: a sparse mixture-of-experts Pro and a distilled dense Flash, tested on needle recall and long-context QA to millions of tokens; closed, and light on architecture and training detail.
• FastV: drops low-attention image tokens after an early layer, with no training.
• Visual-RFT: Visual Reinforcement Fine-Tuning: reinforcement fine-tuning of Qwen2-VL with rule-based rewards for detection and classification.
• SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis: a zero-shot multi-agent workflow with a knowledge check and retrieval, evaluated on a surgical frame benchmark.
• VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning: Qwen2.5-VL-7B trained with GRPO on temporal grounding, tracking and grounded QA.
• InfoSeek: a visual QA benchmark for questions that need outside knowledge about the pictured entity.
• On the Adversarial Robustness of Multi-Modal Foundation Models: white-box APGD attacks with imperceptible perturbations that steer OpenFlamingo's captions and answers.
• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents: a LLaVA-style model that writes tool calls in three fields and gets tool outputs back in the chat.
• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts: draws boxes, masks, arrows and scribbles onto the image before a LLaVA-style encoder reads it.
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving: an MLLM that outputs one speed and one path decision for a motion planner, tested in CARLA.
• ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation: a LLaMA-Adapter fine-tuned to output a contact pixel and gripper directions for simulated pulling tasks.
• Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs: UniME, a two-stage recipe that turns an MLLM into an image-text embedder.
• Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models: 972 harmful seeds rendered in 24 text, image, video and audio combinations, with understanding-aware safety scores.
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding: a training-free KV-cache policy for streaming video that assigns a memory role to each layer.
• Sensory PID: a partial-information split of how video and audio shape omni-modal answers, and PID-based sample weights for vision-language fine-tuning.
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework: one encoder-decoder that writes text, image codes and box tokens from an instruction, with no task heads.
• PaLI: A Jointly-Scaled Multilingual Language-Image Model: a ViT and an mT5 encoder-decoder trained on a multilingual mix of image-text tasks, with the largest model scaling the ViT to 4B parameters.
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering: ScienceQA, 21,208 science questions with lectures and explanations, and answer-first chain-of-thought prompting.
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models: a 188M Q-Former trained in two stages between a frozen image encoder and a frozen LLM.
• Multimodal Chain-of-Thought Reasoning in Language Models: two T5 models, one writing a rationale from text and frozen ViT features, the other answering from the text and the rationale.
• Visual Instruction Tuning: LLaVA, a frozen CLIP encoder joined to Vicuna by one trainable projection, tuned on 158K GPT-4-written instruction samples.
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models: a frozen BLIP-2 vision stack and frozen Vicuna joined by one trainable projection, with a second stage on curated detailed descriptions.
• A Survey on Multimodal Large Language Models: a survey of the encoder, connector and LLM design, the three training stages and hallucination; it runs no experiments.
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic: writes image coordinates as plain numbers in the text, with a frozen CLIP encoder and one linear layer into Vicuna.
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark: 20 temporal video tasks in multiple-choice form, and VideoChat2, a three-stage video model evaluated on them.
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone: the phi-3 family of small models trained on filtered and synthetic data, including the 4.2B vision model phi-3.5-Vision.
• GPT-4o System Card: a closed omni-model's system card, covering speech safety, preparedness scores and third-party tests; no code or weights.
• MedGemma 1.5 Technical Report: MedGemma 1.5 4B adds 3D CT and MRI slice sequences, whole-slide tiles, X-ray localisation and longitudinal chest X-ray comparison.
• ClipCap: CLIP Prefix for Image Captioning: a frozen CLIP embedding mapped to a GPT-2 prefix, with an MLP or transformer mapper trained on captions.
• SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities: speech units added to a LLaMA vocabulary, three training stages, and human-judged examples only.
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding: a training-free sampler that contrasts the logits for the image with those for a noised copy.
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models: a 1000-item test of text reading in multimodal models across five jobs, scored by checking for the answer string.
• GLaMM: Pixel Grounding Large Multimodal Model: a chat model that attaches a mask to each object phrase it names, with the GCG benchmark and the GranD dataset.
• Generative Multimodal Models are In-Context Learners: Emu2, a 37B model that predicts text tokens or image embeddings in one interleaved sequence, with few-shot results and an image decoder.
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning: mixture of LoRA experts for LLaVA, trained one task at a time without replay, with router and expert stabilisation.
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality: a CLIP encoder and a learnable visual abstractor on LLaMA-7B, with a frozen-LLM stage and a LoRA stage.
• Otter: A Multi-Modal Model with In-Context Instruction Tuning: OpenFlamingo-9B tuned on MIMIC-IT with in-context image and instruction examples, training only the resampler and gated attention.
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks: a language-guided tokenizer and a LoRA-tuned Alpaca that writes detection and segmentation as class and location tokens.
• MotionGPT: Human Motion as a Foreign Language: VQ-VAE motion codes added to a Flan-T5 vocabulary, with text-to-motion, motion-to-text, prediction and in-between tasks.
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs: the MMVP benchmark of CLIP-blind image pairs, and interleaved CLIP and DINOv2 tokens for LLaVA.
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces: VSI-Bench of spatial questions over indoor video, with text-only reasoning and cognitive-map probes.
• GeoChat: Grounded Large Vision-Language Model for Remote Sensing: a LLaVA-1.5-style remote-sensing chat model with boxes written as text and task tokens, trained on a dataset built from aerial detection labels.
• Think Then Embed: Generative Context Improves Multimodal Embedding: a multimodal LLM writes a short reasoning text before the embedding is computed, in teacher, small-reasoner and unified variants.
• FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction: a VLA policy with a Qwen2.5-VL supervisor that is queried at gripper changes, plus a fusion of past action predictions.
• Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation: DualVLN, a Qwen2.5-VL planner that outputs pixel goals and a diffusion transformer that outputs 32 waypoints, run asynchronously.
• End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering: VLMnav, a zero-shot navigation policy in which a VLM picks one of several numbered arrows drawn on a depth-checked camera image.
• Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs: a small imitation driver that a frozen multimodal LLM steers with LEFT, MIDDLE or RIGHT hints, tested on a real RC car.
• InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models: route instructions written by a large multimodal model from house-tour videos, checked for consistency, used to pretrain a navigation agent.
• Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces: a LoRA-tuned LLaVA-1.5-7B on a human-annotated question set about robot-human scenes, rated above two prompted models by human judges.
• ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model: a review of dental uses for multi-modal LLMs, with one illustrative pipeline and no evaluation.
• Capabilities of Gemini Models in Medicine: Med-Gemini, Gemini 1.0 and 1.5 models tuned for medicine, with self-trained search, multimodal tuning and long-context prompting; the models are not released.
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models: a LLaMA2-7B VLM that reads mask-pooled region vectors and an optional depth input, trained on OpenImages questions built from 3D scene graphs.
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics: about 3M synthetic spatial questions from 3D scans, used to fine-tune 2D and 3D VLMs, with a non-commercial official generator.