mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

In plain words

Large language models read and write text, but they cannot see. One way to add sight is to hand the picture to a separate vision system and let the language model summarise the result. mPLUG-Owl instead connects a vision model directly to a language model that already exists, and trains the connection in two steps.

In the first step, the picture side learns to describe images in a form the language model understands, while the language model itself is left unchanged. Think of it as teaching a translator to read a new alphabet without changing the language they already speak. In the second step, the language model gets a light set of add-on weights and is trained on ordinary instructions, some of them with pictures attached. That mix is meant to keep its writing skill while teaching it to answer questions about what it sees.

The authors' point is that keeping the big language model mostly intact, and training the picture side and a small add-on separately, gives better answers about images than earlier systems that froze the vision side. The card below gives the details and the places where the paper and its code disagree.

TL;DR

mPLUG-Owl adds image input to LLaMA-7B with a CLIP ViT-L/14 encoder and a learnable visual abstractor that compresses patch features into a short token sequence §3.1. Stage 1 aligns image and caption with the LLM frozen §3.2. Stage 2 fine-tunes LoRA adapters on the LLM with text-only and image-text instructions §3.2. The headline result is author-reported: 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4 §4.2. Stage 1 hyperparameters disagree within the paper, and the repository releases only the stage-2 code.

Problem

Systems that use a language model to call vision tools can answer simple visual questions, but they depend on the tools and may not follow multimodal instructions well §1. End-to-end systems that attach a frozen vision encoder to a language model are cheaper, but the authors argue that a frozen encoder gives weak alignment with few parameters to adapt, and that these systems rarely train on instruction data that mixes text and images §1 §3.1. The paper assumes that a pretrained vision encoder and a pretrained LLM are both available and that instruction data can be collected at scale §3.2 §4.1.

Method

The model has three parts: a visual foundation model f_{\mathbf{V}}, a language foundation model f_{\mathbf{L}}, and a visual abstractor f_{\mathbf{K}} between them §3.1 fig. 2. The vision model turns an image into dense patch features. These are too long for the LLM to process cheaply, so the abstractor summarises them into a set of learnable tokens §3.1. The resulting visual tokens are placed together with the text tokens and fed to the LLM, which generates the response §3.1. fig. 1 compares this design with three earlier end-to-end families §3.1; the paper does not give the number of learnable tokens.

Training follows two stages §3.2 fig. 1.

Stage 1: multimodal pretraining. f_{\mathbf{V}} and f_{\mathbf{K}} are trained while f_{\mathbf{L}} stays frozen. The data are image-caption pairs from LAION-400M, COYO-700M, Conceptual Captions and MSCOCO §4.1. The loss is the language-modelling objective over text tokens only; visual tokens do not enter the loss §3.2.

Stage 2: joint instruction tuning. The whole model is frozen, and LoRA adapters are trained on f_{\mathbf{L}} §3.2. The data come from text-only instruction sets (102k Alpaca, 90k Vicuna, 50k Baize) and 150k multimodal examples from LLaVA §4.1. Each record is turned into a conversation in the Vicuna style, and the loss is computed on the response §3.2. Text-only and multimodal gradients are accumulated over several batches before each update §3.2.

The paper has no numbered equations. The training objective is described in words only §3.2.

# Stage 1: image-text alignment  [§3.2] [§4.1]
# Trainable: f_V (ViT) and f_K (abstractor). Frozen: f_L (LLM).
def pretrain_step(images, captions):
    # Dense patch features, width 1024  [§3.1] [§4.1]
    patches = f_V(images)                    # (B, P, 1024); P not stated
    # Summarise into learnable query tokens at LLM width  [§3.1]
    visual_tokens = f_K(patches)             # (B, Q, d_lm); Q not stated
    # Language-modelling loss on the text tokens only  [§3.2]
    logits = f_L(visual_tokens, captions)
    return next_token_loss(logits, captions)

# Stage 2: joint instruction tuning  [§3.2] [app. A]
# Trainable: LoRA adapters on f_L only. Frozen: f_V, f_K, LLM weights.
def instruction_step(batch):
    # Records without an image carry no visual tokens  [§3.2]
    visual_tokens = None
    if batch.has_image:
        visual_tokens = f_K(f_V(batch.images))
    # Each record is one conversation in Vicuna style  [§3.2]
    logits = f_L(visual_tokens, batch.conversation)
    # Loss only on the assistant response, not on the prompt  [§3.2]
    return masked_next_token_loss(logits, batch.response_mask)

Hyperparameters

The paper's text §4.1 and its appendix table app. A disagree on several stage-1 values. Both are shown. "not stated" means neither source gives the value.

Results

Author-reported. Setup: OwlEval, 82 open-ended questions built on 50 images, rated by hand as A, B, C or D §4.2. mPLUG-Owl is 7.2B parameters §4.1.

Ability accuracy in the ablation tab. 2. Each value is the share of questions where the ability appears in the response, in percent. The per-ability question counts are not stated.

The six column codes mean, in order: following an instruction, perceiving the picture, reading text in the picture, moving knowledge between the image and language, multi-step reasoning, and keeping up a multi-turn dialogue. Their exact definitions are in tab. 1. The ablation is a single run with no variance reported.

Paper vs code

Code checked: X-PLUG/mPLUG-Owl at commit 0f3068fdb47b77aedc71fc39f5735cd7aa35e8f9. The model code for this paper is in the mPLUG-Owl/ folder; the same repository also holds mPLUG-Owl2/ and mPLUG-Owl3/, which are separate works and were not used. That folder's LICENSE is Apache-2.0 LICENSE. The repository's top-level LICENSE is MIT and covers other code; the card uses the Apache-2.0 license of the mPLUG-Owl/ folder.

What matches:

• The vision configuration (24 layers, width 1024, patch 14, image 224) matches §4.1 configuration_mplug_owl.py.
• The stage-2 script sets learning rate 2e-5, weight decay 0.0001 and warmup 50, the same as app. A train_it.sh. It also sets AdamW betas (0.9, 0.999), but the trainer never receives them (see below).
• The OwlEval question file has 82 lines, as the paper states questions.jsonl.

Where they differ:

• Which parts train in stage 2. The paper's abstract and introduce LoRA on the LLM and the abstractor together §1. §3.2 and the released code train LoRA on the LLM only. With --use-lora, every base parameter is frozen, including the abstractor, and LoRA is attached to the q_proj and v_proj layers of the language model train.py.
• LoRA settings. The paper does not state rank, alpha, dropout or target modules. The code defaults are rank 8, alpha 32, dropout 0.05 [code: mPLUG-Owl/pipeline/train.py#L29-L34].
• A second, non-LoRA script. scripts/train_it_wo_lora.sh trains all language-model weights at learning rate 1e-5 [code: mPLUG-Owl/scripts/train_it_wo_lora.sh#L55]. The paper does not describe this variant.
• Batch size. The script defines global_batch_size=256, but the value is never passed to the trainer. With 8 processes and micro-batch 4 and accumulation 1, the effective batch is 32 train_it.sh. The no-LoRA script (micro-batch 1, accumulation 4) also gives 32.
• Training length. The paper says 2,000 steps §4.1. The script sets 3 epochs and computes 4,236 steps in a comment, but num_training_steps is not passed to the trainer train.py. The comment gives 361,481 records. The paper's four sources add up to 392k (102k + 90k + 50k + 150k) §4.1. The data files are not in the repository.
• Optimizer and schedule. The trainer receives only learning rate, warmup, epochs, weight decay, clipping and precision. It does not receive betas, epsilon, minimum learning rate or a scheduler type train.py. The cosine helper get_cosine_schedule_with_warmup is defined but not called from pipeline/ utils.py. The schedule and epsilon the code actually uses are therefore the Trainer's defaults, which this card has not verified against a pinned library version.
• Sequence length. The paper's table says 1,024 for stage 2 app. A. The script passes 2,048 train_it.sh.
• Image augmentation. The code applies a random horizontal flip as well as the random resized crop (scale 0.5 to 1.0) [code: mPLUG-Owl/pipeline/data_utils/processors/caption_processor.py#L27-L28]. The paper mentions only the resize §4.1. RandAugment is off in the config v0.yaml.
• Visual token count. The data loader reserves 65 positions per image, 1 plus 64 xgpt3_dataset.py. The default config has 64 query tokens [code: mPLUG-Owl/mplug_owl/configuration_mplug_owl.py#L226]. The paper does not give the number.
• Stage 1 code is missing. Only the instruction-tuning entry point is in the repository train.py. The stage-2 script loads the checkpoint MAGAer13/mplug-owl-llama-7b-pt [code: mPLUG-Owl/scripts/train_it.sh#L47] and does not say how it was produced.
• Loss on response tokens. The code gives label 1 only to the assistant response and its end-of-sequence token, and label 0 to the prompt [code: mPLUG-Owl/pipeline/data_utils/xgpt3_dataset.py#L192-L213]. This matches §3.2.

No discrepancies were found for the OwlEval evaluation set itself.

Pitfalls

• Stage 1 numbers conflict inside the paper. The text gives learning rate 0.0001, warmup 2k steps, weight decay 0.1 and a batch of 2.1M tokens §4.1. The appendix table gives 2e-4, 375 steps, 0.01 and 4,096 image-caption pairs app. A. The stage-1 code is not released, so an implementer cannot resolve this from the repository. Treat stage 1 as under-specified.
• Abstractor status in stage 2 is contradictory. The abstract and §1 describe training the abstractor with LoRA. §3.2 and the code freeze it. Follow the code if you need to reproduce the released checkpoints.
• LoRA dropout is parsed as an integer. The argument is declared with type=int, so --lora-dropout 0.1 on the command line fails. Only the default 0.05 works train.py.
• Effective batch depends on the GPU count. The scripts hard-code 8 processes train_it.sh. Under that setting the effective batch is 32, not the 256 in the paper.
• Random horizontal flips alter spatial questions. The training transform mirrors images. Questions about left and right, or text printed on an image, can point the wrong way after a flip. The paper does not mention flipping §4.1.
• Visual token count is not stated. The implementer must choose the number of learnable query tokens. The code uses 64 [code: mPLUG-Owl/mplug_owl/configuration_mplug_owl.py#L226]; a comment in the data loader still says 256 xgpt3_dataset.py.
• Abstractor size is not stated. The code default is 6 layers, width 1024, 16 heads, and a linear projection to the LLM width [code: mPLUG-Owl/mplug_owl/configuration_mplug_owl.py#L133] [code: mPLUG-Owl/mplug_owl/modeling_mplug_owl.py#L954]. Pretrained weights are needed to match the paper's model.
• Evaluation is manual and small. OwlEval has 82 questions §4.2. The paper does not state how many raters scored responses, whether they agreed, or when the baseline demos were queried. The baselines come from released code and public demos §4.1 §4.2.
• MM-REACT comparison drops failed cases. For a fair comparison the authors removed MM-REACT's failed predictions before scoring §4.2 app. B. Its headline numbers therefore cover a different set of questions.
• Text-only evaluation uses ChatGPT as judge. The authors say GPT-4 was not available and use ChatGPT as a "suboptimal scorer" §4.3. The scoring scale is not stated. This is a single judge with no variance.
• Ablation denominators are not stated. Percentages in tab. 2 are shares of an unstated count per ability.
• OCR and numbers are weak. The authors report that perception of numbers in images is limited, and that complex OCR scenes are handled only generally §5. Document understanding is described as preliminary and partly failing §5.
• Multilingual ability is not trained for. The paper reports Chinese, French and Japanese responses that arise from LLaMA's text knowledge. It also says the model may fail to answer in the right language §5.

Lineage

• Builds on: LLaMA-7B as the language model §4.1; CLIP ViT-L/14 for the initial vision weights §4.1; LoRA for stage 2 §3.2; the Vicuna conversation format §3.2; modular training ideas from mPLUG, mPLUG-2, E2E-VLP and HiTeA §1.
• Compared against: MiniGPT-4 §4.1; LLaVA §4.1; BLIP-2 §4.1; OpenFlamingo and MM-REACT §4.1.
• Superseded by: not known from content in this wiki. The released repository contains mPLUG-Owl2/ and mPLUG-Owl3/ folders, but this wiki has no cards for those works.