A Survey on Multimodal Large Language Models

In plain words

Large language models read and write text very well, but they cannot see. Vision models see well but reason poorly. A multimodal large language model joins the two: a language model is the "brain", and an image (or sound, or video) encoder turns the other input into something the language model can read.

This survey maps the field as it stood when it was written. It explains the usual three-part design, how these models are trained in stages, how they are tested, and the main problems: they sometimes describe things that are not in the picture, and they are still weak at learning from a few examples or step-by-step reasoning when images are involved.

Read it as a map of the area rather than as a recipe. It does not propose a model of its own, and it does not test anything itself. Its value is in pointing to the methods and datasets to read next.

TL;DR

A survey of multimodal large language models (MLLMs). An MLLM has three parts: an encoder for the non-text input, a language model that starts from pre-trained weights, and a connector between the two §2. Training runs in three stages: pre-training on caption data, instruction tuning, and alignment tuning with RLHF or DPO §3. The survey also covers evaluation §4, hallucination §6, and multimodal in-context learning, chain-of-thought and LLM-aided visual reasoning §7. It reports no new results; its numbers are reproduced from cited papers.

Problem

LLMs can only process discrete text, so they are "blind" to vision, while large vision models see well but lag at reasoning §1. Earlier multimodal work was either discriminative (projecting image and text into one space, for example CLIP) or generative (unifying tasks as sequence-to-sequence, for example OFA) §1. The survey asks how these two lines can be combined into one model that receives, reasons over and outputs multimodal information §1.

The survey gives no problem statement for a single method. Its scope is the whole field, mainly visual and language inputs, with some video and audio §1.

Method

This is a survey, so there is no method to reimplement. The card below gives the general design the survey describes and the training objectives it writes down.

1. Architecture. A typical MLLM has three modules. A pre-trained modality encoder turns the image, audio or video into features. A pre-trained LLM reasons over them. A modality interface connects the two §2 fig. 2. Some models add a generator to output other modalities §2.
2. Modality encoder. Pre-trained encoders are preferred, because a CLIP-style encoder is already aligned to text §2.1. The survey finds that input resolution matters more than parameter size or training data mix, citing an empirical study §2.1. Two ways to raise resolution are direct scaling and patch division, where sub-images and a downscaled full image go through the same encoder §2.1 tab. 1.
3. Pre-trained LLM. Most open LLMs are causal decoders that follow GPT-3; Flan-T5 is the encoder-decoder exception §2.2 tab. 2. Mixture-of-experts LLMs are reported to beat dense counterparts on almost all benchmarks in two cited studies §2.2.
4. Modality interface. Two options. A learnable connector is either token-level (features become extra input tokens; query-based compression as in Q-Former, or a plain MLP as in LLaVA) or feature-level (cross-attention layers or a visual expert inside the LLM) §2.3. The other option is an expert model, which turns the image into text, with information loss §2.3.
5. Pre-training. Next-token cross-entropy on caption pairs, with the image tokens as prefix. Only the caption tokens count in the loss §3.1 tab. 3. Usually the encoder and LLM stay frozen and only the interface is trained; some works unfreeze the encoder §3.1.
6. Instruction tuning. A sample is a triplet of instruction, multimodal input and response. The model predicts the answer given instruction and input eq. 1. The loss is the autoregressive negative log-likelihood of the response eq. 2 §3.2. Instruction data are built by adapting existing datasets, by self-instruction with a strong LLM, or by mixing with text-only conversation data §3.2.
7. Alignment tuning. RLHF has three steps: supervised fine-tuning, a reward model trained on preference pairs eq. 3, and PPO with a per-token KL penalty to a reference model eq. 4 §3.3. DPO skips the explicit reward model and uses a binary-classification loss on preference pairs eq. 5 §3.3.
8. Evaluation. Closed-set benchmarks with fixed answer options, in zero-shot or fine-tuned settings. Open-set chat evaluation by manual scoring, GPT scoring, or case studies §4.
9. Hallucination. Three types: existence, attribute and relationship §6.1. Evaluation with CHAIR, POPE, MME, LLM-judged and GPT-4V-judged methods §6.2. Mitigation before, during, or after generation §6.3.
10. Extended techniques. Multimodal in-context learning by adding demonstrations at inference §7.1. Multimodal chain-of-thought, learned by fine-tuning, few-shot or zero-shot prompting §7.2. LLM-aided visual reasoning, where an LLM acts as controller, decision maker or semantics refiner over tools and vision models §7.3.
11. Open challenges. Long multimodal context, more complicated instruction following, weak multimodal in-context learning and chain-of-thought, embodied agents, and safety. The survey notes that a mainstream way to make high-quality question-answer data is still prompting the closed-source GPT-4V §8.

Pseudocode for the generic forward pass (sections 1 to 4). It is a sketch of the survey's three-module design, not one published model.

# Generic MLLM forward pass: encoder, connector, LLM  [§2] [fig. 2]
# Input:  image (H, W, 3); text_tokens (T,) integer ids
# Output: next-token logits (K + T, vocab_size)

def mllm_forward(image, text_tokens):
    # 1. Encode the non-text input with a pre-trained encoder  [§2.1]
    features = modality_encoder(image)            # (N, d_enc)

    # 2. Map features into the LLM embedding space  [§2.3]
    # Token-level fusion shown: query tokens (Q-Former) or an MLP
    # Feature-level fusion instead lives inside the LLM  [§2.3]
    visual_tokens = connector(features)           # (K, d_llm)

    # 3. Embed the text and join it with the visual tokens  [§2.3]
    text_embeddings = embed(text_tokens)          # (T, d_llm)
    sequence = concat(visual_tokens, text_embeddings)

    # 4. Run the pre-trained LLM over the joined sequence  [§2.2]
    return llm(sequence)

Key equations, as written in the survey (its own notation):

\mathcal{A} = f(\mathcal{I}, \mathcal{M}; \theta) eq. 1

\mathcal{L}(\theta) = -\sum_{i=1}^{N} \log p(\mathcal{R}_i \mid \mathcal{I}, \mathcal{R}_{<i}; \theta) eq. 2

\mathcal{L}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \left[ \log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big) \right] eq. 3

\mathcal{L}(\phi) = -\mathbb{E}_{x\sim \mathcal{D},\, y\sim \pi^{RL}_\phi(y|x)} \Big[ r_\theta(x,y) - \beta\, \mathbb{D}_{KL}\big(\pi^{RL}_\phi(y|x) \,\|\, \pi^{REF}(y|x)\big) \Big] eq. 4

\mathcal{L}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \Big[ \log \sigma\Big( \beta \log \tfrac{\pi^{RL}_\phi(y_w|x)}{\pi^{REF}(y_w|x)} - \beta \log \tfrac{\pi^{RL}_\phi(y_l|x)}{\pi^{REF}(y_l|x)} \Big) \Big] eq. 5

Pseudocode for the instruction-tuning and DPO losses, as the survey writes them.

# Instruction-tuning loss on one sample  [§3.2] [eq. 1] [eq. 2]
# Input:  instruction (text), image (or None), response (N tokens)
# Output: scalar loss (negative log-likelihood of the response)

def instruction_loss(instruction, image, response):
    # Build the prompt from the instruction and the multimodal input  [tab. 5]
    context = build_prompt(instruction, image)
    loss = 0.0
    for i in range(len(response)):
        # Add the negative log-probability of each response token  [eq. 2]
        loss = loss - log_prob(response[i], context, response[:i])
    return loss

# DPO loss on one preference pair  [§3.3] [eq. 5]
# Input:  prompt x; preferred y_w; rejected y_l; beta (scalar)
# Output: scalar loss

def dpo_loss(prompt, preferred, rejected, beta):
    # Log-ratio of policy to reference, for each response  [eq. 5]
    ratio_w = log_prob_policy(preferred, prompt)
    ratio_w = ratio_w - log_prob_ref(preferred, prompt)
    ratio_l = log_prob_policy(rejected, prompt)
    ratio_l = ratio_l - log_prob_ref(rejected, prompt)
    # Binary-classification loss on the gap between the two ratios  [eq. 5]
    return -log_sigmoid(beta * (ratio_w - ratio_l))

Hyperparameters

The survey runs no training of its own. The table lists only values it states. Everything else is not stated.

Results

Author-reported, and none of them are the survey's own. The survey runs no experiments §1. Benchmarks it names, and what it says each measures:

The survey cites several headline results without reproducing their numbers, for example the comparison of GPT-4V and Gemini-Pro on visual reasoning §4. None of those numbers appear on this card.

Paper vs code

No official code. The paper links one GitHub repository (BradyFU/Awesome-Multimodal-Large-Language-Models) in its abstract. Its timeline figure caption points to the same page fig. 1. It is a curated list of papers, not implementation code, so it is not cloned or recorded as code. The survey's own figures and tables are not implemented anywhere. The equations and pseudocode above come from the paper text, not from code.

Pitfalls

• It is a survey, not a method. There is no training recipe, no hyperparameter set and no single model to match. An implementer should read the cited papers for those (for example BLIP-2 and LLaVA, which have cards here), not this card.
• Source version. The text was read from the LaTeX source that fetch downloaded (metadata version v4, updated 2024-11-29). Section and table numbers follow that source. Equation numbers were counted in source order because most equations have no label; check them against the PDF before you cite one.
• Text and table disagree on LAION-5B. The text says 5.85B image-text pairs §3.1; Table 4 says 5.9B tab. 4.
• Priority claim. The authors say this is the first survey of MLLMs §1. That is their claim, and I did not check it.
• Figures. fig. 3 is credited in its caption to another paper §3.2 and is not reproduced. fig. 1 and fig. 2 are the authors' own, but no images are used on this site yet (see CLAUDE.md).
• Expert-model interface is lossy. Turning video into text distorts spatial and temporal relations, as the survey says for VideoChat-Text §2.3. It is a trade-off, not a verdict.
• Resolution claim. "Parameter size and training data composition are of less importance" than resolution §2.1 rests on one cited empirical study (MM1). The survey states it as a general finding.
• Connector choice. The survey reports that for VQA, token-level fusion beat cross-attention in one study, and that cross-attention may need a more involved hyperparameter search §2.3. No general ranking is given.
• Data-quality claims rest on single studies. Lynx on noisy pre-training pairs, Du et al. on task type, and Wei et al. on instruction quality §3.2 are each one study, quoted as reported.
• GPT-4 as a judge. Text-only GPT-4 judges cannot see the image, so the survey calls it questionable as an upper bound §4. GPT-4V judges see the image, but they are still model judges.
• POPE keyword matching. Open answers are turned into yes/no by detecting those words §6.2. A hedged or negated answer can be misread; the survey does not measure this.
• DPO and RLHF sign and scale. Eq. 4 is written as a negative expectation of reward minus β times KL. Check the sign convention against your own code. Eq. 5 is written with whole-response log-probabilities; whether they are summed over tokens or averaged is not stated.
• Pre-training loss mask. Only the caption tokens count in the pre-training loss tab. 3. Whether the image tokens are masked as well is not stated.

Lineage

• Builds on: the survey traces CLIP and OFA as the two earlier multimodal lines §1; Q-Former (BLIP-2) §2.3; LLaVA's linear projection and visual instruction data §2.3 §3.2. Cards: BLIP-2, Visual Instruction Tuning (LLaVA), and for the chain-of-thought lineage in §7.2, Chain-of-Thought Prompting.
• Compared against: not applicable (survey).
• Superseded by: not stated. The survey says it will be kept up to date §1.