Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

In plain words

Many chatbots that can look at pictures use one fixed image encoder, CLIP, that was trained to match pictures with their captions. The authors found pairs of photos that look clearly different to a vision-only model, yet CLIP gives them almost the same internal description. Asking simple questions about those pairs, such as which way something faces or how many objects there are, exposes the gap: most of the tested chatbots do badly, while people answer almost every question right.

The paper also sorts these failures into nine kinds of visual detail and tests whether bigger CLIP models fix them. Most do not. As a first fix, the authors feed the chatbot a second set of image features from the vision-only model, DINOv2, interleaved with the CLIP features. This helps the model see the details, and the instruction-following ability stays about the same.

TL;DR

• Open multimodal LLMs mostly use a frozen CLIP image encoder, and that encoder misses simple visual details §1.
• The MMVP benchmark (150 image pairs, 300 questions) and the MMVP-VLM benchmark (nine patterns) expose the gap. Author-reported.
• Interleaved-MoF adds DINOv2 tokens between CLIP tokens and roughly doubles the visual token count; it raises MMVP for LLaVA and LLaVA-1.5 with a small change elsewhere tab. 3.
• Code and configs are in the repository linked above; the released scripts cover the LLaVA-1 setting only.

Problem

Models that pair a vision encoder with an LLM answer simple questions about image details wrongly, even when they describe scenes well §1 §2.3. The authors ask whether the shortfall starts in the pretrained vision encoder. Without a benchmark that targets those details, the question cannot be tested, because standard benchmarks such as ImageNet zero-shot accuracy do not track it §3.2 app. C.

Method

Stage A: find CLIP-blind pairs §2.1. The authors take candidate images from ImageNet and LAION-Aesthetics §2.1. A pair is kept when its CLIP-ViT-L-14 embeddings have cosine similarity above 0.95 and its DINOv2-ViT-L-14 embeddings have cosine similarity below 0.6 §2.1. The paper does not say how candidate pairs are enumerated. The pseudocode below shows the filter only.

Stage B: write the benchmark §2.2. The authors manually inspect each of 150 pairs, find the visual difference CLIP overlooks, and write two questions per pair, giving 300 questions §2.2 fig. 2. A pair counts as correct only when both of its questions are answered correctly §2.2 fig. 3. Questions are asked independently, without chat history §2.3.

Stage C: name the failure patterns §3.1. GPT-4 is prompted with the benchmark's questions and options and asked to propose general patterns that an image-embedding model struggles with §3.1. The result is nine patterns: facing and direction, whether a feature is present, the state of an object, counting, spatial relations, color, physical structure, printed text, and the camera's viewpoint §3.1 app. D. The VLM variant (MMVP-VLM) uses 15 image-text pairs per pattern §3.2.

Stage D: test CLIP-based models §3.2 tab. 1. The authors evaluate OpenAI CLIP, SigLIP, DFN, MetaCLIP and EVA models on MMVP-VLM, and ask whether scale and resolution help per pattern §3.2 tab. 1.

Stage E: Mixture-of-Features (MoF) §4. Setting: LLaVA with a CLIP encoder and an adapter to the LLM, trained with the same recipe as LLaVA §4.1 app. A. Two variants:

• Additive MoF. The CLIP and DINOv2 features are mixed linearly with a coefficient alpha before the adapter §4.2 fig. 7. Tested DINOv2 shares are 0, 0.25, 0.5, 0.625, 0.75, 0.875 and 1.0 §4.2 tab. 2. Visual grounding rises with the DINOv2 share, but instruction following falls §4.2.
• Interleaved MoF (I-MoF). Each encoder is passed through its own adapter, then the two token sequences are interleaved in their original spatial order and fed to the LLM §4.3 fig. 7. The authors report that this improves MMVP while keeping LLaVA-bench scores about the same §4.3 tab. 3.

Figures 1–12 are described in the text; no image is embedded in this card.

# CLIP-blind pair mining  [§2.1]
# Input:  candidate images (ImageNet, LAION-Aesthetics)
# Output: pairs that CLIP merges but DINOv2 separates

def mine_clip_blind_pairs(images):
    # Embed each image with both reference encoders  [§2.1]
    clip_vectors = clip_vit_l_14(images)
    dino_vectors = dinov2_vit_l_14(images)
    # How candidate pairs are enumerated is not stated  [§2.1]
    pairs = []
    for image_a, image_b in candidate_pairs(images):
        # CLIP must see the two images as nearly identical  [§2.1]
        clip_similar = cosine(clip_vectors[image_a],
                              clip_vectors[image_b]) > 0.95
        # DINOv2 must see them as clearly different  [§2.1]
        dino_different = cosine(dino_vectors[image_a],
                                dino_vectors[image_b]) < 0.6
        if clip_similar and dino_different:
            pairs.append((image_a, image_b))
    return pairs

# Interleaved MoF token merge  [§4.3]
# [code: LLaVA/llava/model/llava_arch.py#L232-L237]
# Input:  clip_tokens (N, d), dino_tokens (N, d), both already projected
# Output: merged (2N, d), placed where one image's tokens would go

def interleave_tokens(clip_tokens, dino_tokens):
    # One CLIP token and one DINOv2 token per patch position  [§4.3]
    # (N patches per encoder: 256 at 224 px, 576 at 336 px  [tab. 3])
    merged = empty(2 * N, d)
    # Even slots take CLIP tokens in raster order  [§4.3]
    merged[0::2] = clip_tokens
    # Odd slots take DINOv2 tokens in the same order  [§4.3]
    merged[1::2] = dino_tokens
    return merged

Hyperparameters

Training follows LLaVA (stage 1: projector pretraining on CC595k; stage 2: instruction tuning) and LLaVA-1.5 app. A. Hardware: at most 8 NVIDIA A100 GPUs app. A. The paper's table is in tab. 4; code values are in the LLaVA folder of the repository.

Results

Author-reported. Setup: LLaVA and LLaVA-1.5 with Vicuna, evaluated on MMVP (300 questions), POPE, LLaVA-bench, MM-Vet and other benchmarks §4 app. E.

On the wider benchmarks, the authors call I-MoF "comparable on most" app. E.2. Its 224² row is lower than LLaVA-1.5 on MMBench (61.6 vs 67.7), TextVQA (55.3 vs 61.3) and VQA-v2 (77.3 vs 80.0) tab. 7.

Paper vs code

The repository is tsb0601/MMVP at commit 763500597e65c3446f09047837ceda76f4e264bf. The owner matches the first author's homepage linked in the README. It is an official repository. It contains LLaVA-based training scripts, a fork of LLaVA under LLaVA/, and evaluation scripts for MMVP and MMVP-VLM README.md. Benchmark data and MoF checkpoints are on Hugging Face and are not in the repository.

Differences between paper and code:

1. LLaVA-1.5 stage 1 learning rate. The paper says 2e-3 app. A tab. 4. The LLaVA-1.5 pretrain script uses 1e-3 pretrain.sh.
2. LLaVA stage 2 epochs. The paper says 3 app. A tab. 4. The MoF finetune script runs 1 epoch finetune.sh.
3. LLaVA batch sizes. The paper says 128 for both stages app. A tab. 4. With eight GPUs the scripts give 256 for stage 1 and 176 for stage 2 pretrain.sh finetune.sh.
4. Resolution. Several LLaVA rows in the paper are at 224² tab. 3. Every released script uses clip-vit-large-patch14-336 pretrain.sh finetune.sh. The 224² rows are not reproduced by the shipped scripts.
5. Additive MoF. No code for it was found. The only DINOv2 path in llava_arch.py is the interleaved one llava_arch.py.
6. DINOv2 adapter source. The finetune script loads a DINOv2 adapter checkpoint from a placeholder path finetune.sh. The shipped pretrain.sh has no DINOv2 flag pretrain.sh, so the stage that trains that adapter is not in the repository.
7. LLaVA-1.5 scripts contain no DINOv2 path. The scripts/v1_5/ files have no DINOv2 flags. Interleaved-MoF therefore runs only through the LLaVA-1 scripts.
8. License. The README says MIT and links LICENSE README.md. The repository has no root LICENSE file. The only license file is LLaVA/LICENSE, which is Apache-2.0. The code_license field records the README claim as stated.
9. Paper details the code does not settle. The paper does not say whether DINOv2 is frozen. The code freezes it dino_encoder.py. It also does not say DeepSpeed stage or the LLaVA-1.5 GPU count; the scripts leave both unset or placeholder.

Pitfalls

• Pair mining is underspecified. The paper gives the two thresholds (CLIP similarity above 0.95, DINOv2 similarity below 0.6) but not how candidates are enumerated §2.1. Thresholds depend on the exact reference models; re-mining with other encoders gives a different pool.
• The benchmark is hand-made. Pairs and questions were written by hand §2.2. The paper reports no inter-annotator agreement and no question validation beyond the human study app. B.4.
• Pattern labels rest on GPT-4 output. The nine patterns come from GPT-4 prompting §3.1. How each question was assigned to a pattern is not stated.
• Chance is 25%, not 50%. Scoring a pair as correct needs two independent correct answers §2.2, so random guessing scores 25% §2.3. Per-model values are only in a figure fig. 4; the in-text 38.7% for GPT-4V is from the ablation app. B.3.
• Closed-model results are time-bound. GPT-4V was run via ChatGPT in October and November 2023, Gemini-Pro via Vertex AI in December 2023, and Bard and New Bing in September and October 2023 app. B.1. Mini-GPT4 answers were prefixed with an instruction to avoid hallucination app. B.1 footnote.
• Interleaving doubles the visual tokens. I-MoF uses 512 tokens at 224² against 256 for LLaVA, and 1152 against 576 at 336² tab. 3 tab. 7. The LLM sees twice as many visual positions, which raises prefill cost. The paper's control, plain resolution increase, gives no gain §4.3.
• Both encoders need the same patch grid. The merge assumes equal token counts and pairs them by index llava_arch.py. At 224 px ViT-L/14 gives 256 patches; at 336 px CLIP gives 576. The DINOv2 tower hard-codes num_patches to 256 dino_encoder.py. It was not checked whether that value is used at 336 px.
• Different feature layers. CLIP takes hidden layer −2 clip_encoder.py. DINOv2 takes the final layer before its last normalisation (x_prenorm) dino_encoder.py. The paper says nothing about either choice. Both towers drop the CLS token and keep patch tokens.
• Same preprocessing for both encoders. One images tensor, produced for CLIP, is passed to both encoders llava_arch.py. DINOv2 therefore sees CLIP's normalisation and input size.
• Weights are fetched unpinned. The DINOv2 tower is loaded through torch.hub from facebookresearch/dinov2 with no version pin dino_encoder.py. It runs in fp16 at inference builder.py.
• Instruction following is traded away. Higher DINOv2 shares lower the LLaVA-bench score tab. 2. I-MoF avoids that at 224² tab. 3, but the paper's wider benchmark rows show losses on some tasks tab. 7.
• Correlation is over nine values. The per-model Pearson coefficients app. C are computed across the nine visual patterns §3.3, so the sample is small.

Lineage

• Builds on: LLaVA and LLaVA-1.5 (training recipe and benchmarks) §4.1; CLIP (ViT-L/14, the encoder under test) §1; DINOv2 (the vision-only reference and MoF partner) §2.1.
• Method idea: "erroneous agreements" between two encoders, after an earlier study of text encoders in text-to-image models §1.
• Compared against: CLIP-based models (OpenAI, SigLIP, DFN, MetaCLIP, EVA) on MMVP-VLM tab. 1; GPT-4V, Gemini, Bard, InstructBLIP, Mini-GPT4 on MMVP §2.3.
• Additional vision-only backbones for I-MoF: MAE and MoCov3 app. E.1 tab. 6.