MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

In plain words

Most tests for AI systems that read both pictures and text ask about a single still image. Those tests cannot tell whether a model understands what happens over time, such as which way something moved or what happened before an action. This paper builds a test for that. It takes questions that already exist for still images and rewrites them so the answer only shows up across several frames of a short clip. The clips and their ground-truth labels come from video datasets that already exist, so no one had to label new footage by hand. Every question is multiple choice, which makes the scoring simple and avoids having another language model judge the answers.

The authors also train a video chatbot in stages, first teaching it to link frames to text, then connecting it to a language model, and last tuning it on a large mix of instructions. They then score that chatbot on the new test. The idea is useful for anyone who needs a fair way to check whether a video model is reading time, not just one frame.

TL;DR

• MVBench defines 20 temporal video tasks and builds 200 multiple-choice questions per task from 11 public video datasets §3.1 §3.2.
• Options are extracted by ending the prompt with a fixed answer prefix, so every reply yields an option letter §3.3 tab. 9.
• VideoChat2, a Vicuna-7B model trained in three stages, scores 51.1% average accuracy against 35.5% for VideoChat (author-reported) tab. 2.
• Results depend on the LLM, frame count and resolution; GPT-4V was run at a different resolution tab. 2. Check the training config before reuse (see Paper vs code).

Problem

Image benchmarks measure spatial understanding only. Earlier video benchmarks either cover very basic video tasks, focus on one domain or scene type, or need expensive manual annotation §1. Scoring open-ended video answers with an LLM also adds bias, so the authors want labelled answers and multiple choice §1 §3.2. The benchmark is meant to need the whole clip, not one frame §1 fig. 1.

Method

Benchmark: MVBench §3

1. Task definition by static-to-dynamic conversion §3.1. The authors start from 9 spatial task families of image benchmarks (MME and MMBench are the references) and rewrite each into a temporal task that a single frame cannot solve. The result is 20 tasks in nine groups: action (5), object (3), position (2), scene (1), count (2), attribute (2), pose (1), character (1), cognition (3) §3.1 tab. 1. Example: "Is the man on the stage?" becomes "What direction is the man moving?" fig. 1.
2. Source selection §3.2. Eleven public video datasets are used, covering first- and third-person views and indoor and outdoor scenes §3.2. Clips are kept at about 5 s to 35 s; very short clips (little motion) and very long ones are dropped §3.2. For STAR, clip start or end points are shifted at random; for CLEVRER, questions needing more than 10 conditions are removed §3.2.
3. Question and options §3.2 fig. 2. ChatGPT writes a question for each video from the task definition. Options come from templates on the ground truth where possible; for Unexpected Action an LLM converts open-ended QA into multiple choice §3.2. Each item gets 3 to 5 options, shuffled; an LLM makes option lengths similar so length does not reveal the answer §3.2. The paper produces 200 items per task §3.2.
4. Evaluation prompt §3.3. A system prompt asks the model to attend to the cause, sequence, object details and movements, and the actions and poses of people §3.3. The question lists options in parentheses, and the prompt ends with the answer prefix "Best Option: (" so the reply starts with the option letter §3.3 tab. 9.

# MVBench item construction  [§3.1] [§3.2]
# Input:  public video annotations from 11 datasets
# Output: multiple-choice QA items, 200 per task

def build_item(video, annotation, task):
    # 1. Keep clips of about 5 s to 35 s  [§3.2]
    if not 5 <= duration(video) <= 35:
        return None
    # 2. Ask an LLM for a question from the task definition  [§3.2]
    question = llm_question(task, annotation)
    # 3. Build options from ground truth, or with an LLM for some tasks  [§3.2]
    options = build_options(task, annotation)
    # 4. Sample 3 to 5 options and make their lengths similar  [§3.2]
    options = equal_length(sample_options(options, low=3, high=5))
    # 5. Shuffle the option order  [§3.2]
    return shuffle(question, options, answer(annotation))

# Scoring one item  [§3.3] [tab. 9]
# Input:  frames (16, 224, 224, 3); question text; options
# Output: predicted option letter

def score_item(model, frames, question, options):
    # 1. Put the options in parentheses in the question  [§3.3]
    prompt = question_with_options(question, options)
    # 2. Append the answer prefix so the reply starts with the option  [§3.3]
    reply = model.generate(frames, prompt + "Best option: (")
    # 3. The option letter is the first character of the reply  [§3.3]
    return reply[0]

VideoChat2 baseline §4

VideoChat2 pairs a video encoder with a Q-Former (a small transformer that compresses visual tokens into a fixed number of query tokens) and a Vicuna-7B LLM §5 §4.2. It is trained in three stages §4.2 fig. 4.

• Data §4.1 fig. 3. About 2M samples from 34 datasets, in six groups: conversation, simple captions, detailed captions, VQA, reasoning and classification. Image and video data are both used. Instruction texts were written by ChatGPT from dataset and task descriptions §4.1.
• Stage 1, vision-language alignment §4.2. The visual encoder is frozen and the Q-Former is trained with three losses: contrastive (VTC), matching (VTM) and caption generation (VTG). The encoder is UMT-L instead of the BLIP-2 encoder §4.2.
• Stage 2, vision-language connection §4.2. The encoder is unfrozen. A linear projection feeds the query tokens into the LLM, trained with the VTG loss on more image and video captions §4.2.
• Stage 3, instruction tuning §4.2 app. A. LoRA is added to the frozen LLM. The encoder and Q-Former are tuned with the VTG loss. Instructions, not questions, are fed into the Q-Former §4.2 app. B.

# VideoChat2 progressive training  [§4.2] [app. A]
# Input:  image and video captions; then instruction data  [§4.1]
# Output: video MLLM with encoder, Q-Former and LoRA adapters

def train_stage1(encoder, qformer, caption_pairs):
    # Freeze the visual encoder; train the Q-Former only  [§4.2]
    freeze(encoder)
    # Losses: contrastive, matching and generation  [§4.2]
    for batch in caption_pairs:
        loss = vtc(batch) + vtm(batch) + vtg(batch)
        update(qformer, loss)
    return encoder, qformer

def train_stage2(encoder, qformer, projection, llm, caption_pairs):
    # Unfreeze the visual encoder; add a linear projection  [§4.2]
    unfreeze(encoder)
    # Feed projected query tokens and text into the LLM; VTG loss  [§4.2]
    for batch in caption_pairs:
        loss = vtg_with_llm(encoder, qformer, projection, llm, batch)
        update(encoder, qformer, projection, loss)
    return encoder, qformer, projection

def train_stage3(model, instruction_data):
    # Add LoRA to the frozen LLM; encoder and Q-Former stay trainable  [§4.2]
    add_lora(model.llm)
    # Instruction text goes into the Q-Former; the question does not  [§4.2]
    for sample in instruction_data:
        loss = vtg_loss(model, sample)
        update(model, loss)
    return model

Hyperparameters

Stage values are from the paper's appendix table app. A. "Code" columns describe the official configs, see Paper vs code.

Other settings: LLM is Vicuna-7B v0 by default; visual encoder UMT-L; Q-Former initialised from BERT-base §5.

Results

Author-reported. Setup: MVBench, 20 tasks, 200 items each §3.2. Video models take 16 frames unless noted; image models take 4 frames tab. 2. Most rows use LLaMA-1-based LLMs; the Mistral row is a different LLM and training mix §5.1 tab. 2. Hardware for the paper's runs: not stated in the main text.

MVBench accuracy per task tab. 2

Headline claims §5.1 tab. 2: VideoChat2 (Vicuna) is over 15 points above VideoChat (51.1 vs 35.5). VideoChat2 (Mistral) is 16.9 points above GPT-4V (60.4 vs 43.5). VideoChat2 with blank video and no LoRA ("text-only") scores 34.7, close to VideoChat's 35.5 §5.1.

Video conversation (ChatGPT-judged) §5.2 tab. 3

Zero-shot video QA (accuracy / score) §5.2 tab. 4

Ablations (MVBench average) §5.3 app. B

Instruction data tab. 5:

Encoder and LLM tab. 6:

Training method tab. 7: freezing encoder and Q-Former everywhere gives 38.5. Tuning the Q-Former in Stages 2 and 3 gives 47.0. Tuning the encoder in Stage 2 as well (encoder frozen in Stage 3) gives 47.5 tab. 7 row 3. Tuning encoder and Q-Former in Stages 2 and 3 gives 51.1 (the default).

System prompt tab. 8: 49.9 (short prompt), 50.5 (adds cause and sequence), 51.1 (adds people's actions and poses; default).

Answer prompt tab. 9: without the prefix, VideoChat's reply contains an option only 78.2% of the time (avg 22.8); with "Best option: (" every reply does (avg 35.5). VideoChat2 goes from 50.1 to 51.1.

Paper vs code

Code is at video_chat2/ in the repo at commit 4dd8210b, MIT license LICENSE. Code paths below are relative to the repo root.

• Stage 2 input frames. Paper: 4 frames §5 app. A. Code: 8 frames config_7b_stage2.py.
• Stage 2 max text length. Paper: 32 app. A. Code: 512 config_7b_stage2.py.
• Stage 1 warmup. Paper: 1 epoch app. A. Code: 0.2 epochs config_7b_stage1.py.
• Batch size. Paper: 2048 / 512 / 128 app. A. Code: 128 / 4 / 4 per GPU config_7b_stage1.py config_7b_stage2.py config_7b_stage3.py. The per-GPU reading follows the config utility, which sets the global batch as batch size times world size config_utils.py. With the single-node scripts (8 GPUs) the global batch is 1024 / 32 / 32, not 2048 / 512 / 128 run_7b_stage1.sh. The slurm stage 1 script uses 4 nodes of 8 GPUs slurm_run_7b_stage1.sh. The paper does not give the GPU count it used.
• Stage 1 pretrained encoder. Paper: UMT-L §4.2. Code loads an external checkpoint named l16_25m.pth for the encoder config_7b_stage1.py. The repo does not say what that checkpoint is.
• Evaluation frames. Paper: 16 frames §5. Code training configs set test frames to 8 config_7b_stage3.py. The MVBench notebook uses 16 mvbench.ipynb.
• Answer prefix. Paper: "Best Option: (" §3.3. Demo notebook: "Best option:(" (no space) demo_mistral.ipynb. Check which string you match on; the paper is the stated source.
• Matching items. Stage 3 LR (2e-5), epochs (3), warmup (0.6), LoRA rank 16 / alpha 32 / dropout 0.1 and Q-Former drop path (0.2) match the paper app. A config_7b_stage3.py config_7b_stage3.py.
• Stage 1 freeze. Paper freezes the encoder in Stage 1 §4.2. The Stage 1 config has no freeze flag config_7b_stage1.py. Not verified in the model code.

The repo also has Mistral and Phi variants; they were not compared with the paper.

Pitfalls

• Frames and resolution differ across rows. Image models use 4 frames; GPT-4V uses 16 frames at 512×512; the others use 224×224 tab. 2. The GPT-4V gap is not a like-for-like comparison.
• LLM and data differ across rows. The headline 51.1 is Vicuna-7B v0 (LLaMA-1-based). The 60.4 Mistral row is trained with SMiT instructions, which the paper names §5.1 but does not describe in the main text. Do not compare the two as the same model.
• Text vs table on character tasks. The text says the model is weaker than text-only on "position, count, and character" §5.1. Table 2 has Character Order at 36.5 (VideoChat2) vs 36.0 (text-only), so that row does not support the claim. Moving Direction (23.0 vs 25.5), Action Localization (23.0 vs 27.0) and Action Count (39.0 vs 41.5) do tab. 2.
• LLM-judged scores. Video conversation scores are judged by ChatGPT §5.2 tab. 3. The paper argues against LLM scoring for MVBench §1, so keep the two kinds of number apart.
• Generated items. ChatGPT writes the questions and some options, and an LLM equalises option lengths §3.2. The paper does not state a human check of the generated items. Answer-key correctness depends on the source annotations.
• Data licences. The benchmark is distributed via Hugging Face [code: video_chat2/MVBENCH.md#L9]. The 11 source video datasets have their own licences. Not checked here; check before redistributing the videos.
• Option extraction is exact, not correct. The 100% hit ratio tab. 9 means the option letter appears, not that it is right. "Not sure" options exist for some tasks §3.2.
• Random vs fixed frame sampling. Training samples frames at random app. A config_7b_stage3.py; evaluation uses middle frames config_7b_stage3.py.
• Batch size. Batch size in the paper's table is ambiguous until the GPU count is known (see Paper vs code).
• Costs. Stage 1 runs 10 epochs on about 25M pairs §4.2 app. A. Stage 2 adds 12M captions with the LLM in the loop §4.2. Extra query tokens increase the Q-Former size app. B.
• Reproducibility. The stage 1 encoder checkpoint is external (see Paper vs code); stage 1 cannot be reproduced from the repo alone.

Lineage

• Builds on: UMT-L visual encoder §4.2; the Q-Former and its three losses from BLIP-2 §4.2; instruction-into-Q-Former from InstructBLIP §4.2; LoRA app. A; the M3IT data format §4.1; VideoChat and VideoChatGPT for instruction data and comparisons §4.1 §5.2; MME and MMBench for the static task list §3.1.
• Compared against: VideoChat, VideoChatGPT, VideoLLaMA, mPLUG-Owl, LLaMA-Adapter, Otter, MiniGPT-4, InstructBLIP, LLaVA and GPT-4V tab. 2.
• Superseded by: not stated in this content.