Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

In plain words

Vision-language models can look at a photo and answer simple questions about it. This paper asks whether they can also answer questions that need knowledge beyond the picture. Take a photo of a church: the building is visible, but when it was built is not.

The authors build InfoSeek, a benchmark of such questions about real-world things: landmarks, animals, food, aircraft and others. It has a small part written by people and a large part generated automatically from facts in Wikidata. They test models in two ways. In the first, the model answers from the image and question alone. In the second, the system first names the thing in the picture and then reads the answer from Wikipedia. Off-the-shelf models do poorly. Fine-tuning on the benchmark helps, and better recognition of the thing in the picture helps most.

TL;DR

• InfoSeek is a visual question answering benchmark for questions that need outside knowledge about the pictured entity §1 §3.
• It has 8.9K human-written questions and about 1.35M questions generated from Wikidata with templates §3.1 §3.2 tab. 2.
• Zero-shot, the pre-trained models score low. After fine-tuning, PaLI-X is the best end-to-end model (22.1 overall on Wikidata, author-reported) §5.1 tab. 4.
• Adding a visual entity recognizer and a Wikipedia reader helps most on the human split (18.2 with FiD). Perfect entity recognition would reach 45.6 there §5.2 tab. 5.

Problem

A photo of a church shows the building but not its construction date §1. Pre-trained vision-language models are tested mostly on questions answered from the image. Common VQA sets such as VQA v2 ask about attributes and objects that need no outside facts §1. Knowledge-based sets have their own gaps. In OK-VQA, the authors found 70.8% of sampled questions answerable without a search engine §2. In ViQuAE, a question-only PaLM (540B) reaches 31.5% accuracy, so many questions can be answered without the image §2 tab. 1. The authors argue that no existing set covers info-seeking questions about a wide range of visual entities §2.

Assumptions: the evaluation targets fine-grained knowledge about a visual entity, not common sense. The No-KB protocol assumes the model stores that knowledge in its weights §4.

Method

Dataset (InfoSeek). Images come from nine image classification and retrieval datasets, linked to Wikipedia articles through the OVEN annotations §3 app. A.4.

1. Images. Landmarks (17%), animals (13%), food (5%), aircraft (3%) and others §3.
2. Human part (InfoSeek-Human, 8.9K). Annotators write 3 to 5 questions per entity from a picture, a 15-word description and Wikipedia section titles, without seeing the answer §3.1. A second group finds a short answer span in the entity's Wikipedia article and labels the question as Time, Numerical or String §3.1 app. A.1. Experts found 95% of sampled answers correct app. A.1.
3. Automatic part (InfoSeek-Wikidata, about 1.35M). Wikidata triples for 300 relations are turned into questions by human-written templates §3.2. Pairs are kept only if the answer appears in the entity's Wikipedia article, and subsampled to balance entities and relations §3.2 app. A.2. Entities: 11,481 in Wikidata, 806 in Human tab. 2.
4. Splits. The evaluation has two parts. Unseen Question holds out QA pairs of seen entities. Unseen Entity holds out whole entities §3.3.

Metrics. Time and String answers are scored by exact match against several references (after normalization). Numerical answers are scored with relaxed accuracy: a single reference value gets a ±10% band, and a predicted range counts as correct if its intersection-over-union with the reference is at least 0.5 §3.3 app. A.3. The overall score is the harmonic mean of the Unseen Question and Unseen Entity accuracies §3.3 app. A.3. The implementation is in the released code and is compared with the paper under Paper vs code.

Protocols. No-KB: the model gets the image and question and predicts the answer directly §4. With-KB: a visual entity recognizer first links the image to a 100K-entry Wikipedia knowledge base, then a language model or reader answers using that entity §4.

No-KB models §4.1. Random and majority-answer baselines. A question-only PaLM 540B with 5 in-context examples §4.1. BLIP-2 and InstructBLIP with frozen ViT-g and Flan-T5-XXL, where only the Q-Former is fine-tuned on InfoSeek-Wikidata §4.1. PaLI-17B and PaLI-X, fine-tuned on InfoSeek-Wikidata because they have minimal zero-shot performance §4.1.

With-KB models §4.2. Entity recognition: a fine-tuned CLIP ViT-L/14 scores KB entries against the image and question, following the OVEN CLIP2CLIP setup, and keeps the top k=5 §4.2. Reader: PaLM 540B with 5-shot prompting on the predicted entity name, or Fusion-in-Decoder (FiD) reading 100 Wikipedia passages (the top 20 for each of the 5 entities) §4.2. An oracle variant uses the gold entity as an upper bound tab. 5.

# With-KB pipeline (CLIP -> FiD)  [§4.2] [app. B]
# Input:  image, question, Wikipedia KB (100K entries)
# Output: answer string

def answer_with_kb(image, question, kb, reader):
    # Score KB entries against image plus question  [§4.2]
    ranked = clip_rank(image, question, kb)
    # Keep the five best entities  [§4.2]
    top_entities = ranked[:5]
    # Take the top 20 Wikipedia passages for each entity  [§4.2]
    passages = retrieve_passages(top_entities, per_entity=20)
    # FiD reads all 100 passages and writes the answer  [§4.2]
    return reader.generate(question, passages)

Hyperparameters

The appendix table below lists the fine-tuning settings for the models that were fine-tuned, as reported app. B. Each cell is the value stated there; "-" means the appendix gives none.

app. B (fine-tuning hyperparameter table).

Other settings stated in the appendix app. B:

• PaLI-17B and PaLI-X: prompt "answer in en: [question] <extra_id_0>". Pre-trained on WebLI with 1 billion image-text pairs §4.1.
• BLIP-2 and InstructBLIP: prompt "Question: [question] Short answer:", using the LAVIS library, length penalty -1. Early stopping on validation performance app. B.
• OFA-large: 20k steps, beam size 5 app. B. Reported 11.7 (Wikidata) and 4.0 (Human) app. B. Not in the main table.
• mPLUG-owl: 10k steps, learning rate 2e-4, batch size 1, LoRA. Reported 7.7 (Human) app. B. Not in the main table.
• PaLM Q-only: 5-shot, prompt "question: ... answer: ..." pairs app. B.
• FiD: T5-large, two stages. First on Natural Questions for 10k steps, then on InfoSeek for 200 steps, batch size 64. Input is 100 passages of at most 192 tokens. Synthetic training triples pair the oracle entity's passage with passages of linked entities from EntLinker app. B.
• Input format for FiD: "question: This is [Entity]. [Question]. context: [Passage]" app. B.

Results

Author-reported. Setup: models fine-tuned on InfoSeek-Wikidata training data (plus InfoSeek-Human where stated), evaluated on the two splits in §3.3. The main table does not name its split. Its numbers differ from the validation table in app. C, so it is presumably the test set, but the paper does not say so.

No-KB, fine-tuned tab. 4 (scores in %):

tab. 4. The Prior row's VQAv2 figure is marked in the paper as taken from another work. The table's "12B" is written as 12.1B in app. B. OK-VQA and VQAv2 are reference rows, not InfoSeek results.

With-KB tab. 5 (overall, %):

tab. 5. The entity-recognition column is the paper's accuracy for CLIP (22.2) and oracle (100) tab. 5.

Per question type, Wikidata unseen-question validation split tab. 6 (Time / Numerical / String, %):

tab. 6. The authors report that end-to-end models are weak on Time and Numerical questions §5.2.

Other headline results (author-reported):

• Zero-shot (before fine-tuning), Wikidata / Human overall: PaLI-17B 2.0 / 1.7, PaLI-X 2.0 / 1.6, BLIP2 11.3 / 8.6, InstructBLIP 7.4 / 7.2 fig. 3. The plotted values come from the figure source. The paper's claim that fine-tuning elicits knowledge rests on the jump from these to tab. 4 §5.1.
• Generalization to unseen entities: fine-tuned BLIP2 raised accuracy on one unseen entity from 18% to 92% on country location §5.1.
• Real-world photos (90 questions on 30 author-taken images, outside the InfoSeek training data): PaLI-17B 22.2%, PaLI-X 38.9% §5.1.
• Head vs tail entities: on the most popular entities the pipelines beat PaLI by a large margin; on the least popular, PaLI-17B beats the pipelines §5.2.
• Full validation and With-KB tables: app. C (Wikidata validation table and full With-KB table).

Paper vs code

Checked: the paper's body has no code link. Its footnote links a project page, which lists two GitHub repositories: edchengg/infoseek_eval, labelled "Contributed Code", and open-vision-language/infoseek, labelled "Dataset" (not cloned). The first is owned by the first author and is the code cited here. Its README calls it a "Community Contributed Repository". Commit f43d0c68 (MIT license, copyright 2023 edchengg).

The repo contains only the evaluation script, BLIP-2 and InstructBLIP fine-tuning and zero-shot inference (LAVIS). It has no code for PaLI, PaLM, FiD, CLIP entity recognition, OFA or mPLUG-owl.

Discrepancies:

• Metric for String and Time. The paper says VQA accuracy is used for these types, and that Time answers get a one-year tolerance app. A.3. The code scores String and Time by normalized exact match, taking the maximum over references, not VQA accuracy infoseek_eval.py infoseek_eval.py. The Time function's docstring says "+/- one year" but its code is exact match only infoseek_eval.py. Any year tolerance would have to be in the reference lists, and those files are not in the repo.
• Overall score. The paper says overall = harmonic mean of split accuracies §3.3. The code agrees infoseek_eval.py infoseek_eval.py. A split's own score is the pooled match rate over all question types, weighted by example count, not the mean of the three type scores infoseek_eval.py.
• Numerical tolerance. Single reference values get ±10% app. A.3, matching the code infoseek_eval.py. Ranges are correct if fully inside the reference or IoU ≥ 0.5 app. A.3 infoseek_eval.py. The box in app. A.3 lists containment as a correct case, so the code matches the box. The body text mentions only the IoU rule.
• Reversed prediction range. If the model writes a reversed range such as "11-9", the parser returns the first number alone, not a range infoseek_eval.py. The paper does not mention this.
• BLIP-2 optimizer and schedule. The paper's table says Adam, batch 16, 400 steps, constant LR 5e-5 app. B. The code uses AdamW at LR 5e-5 run_training_lavis.py, batch 32 with 4 gradient-accumulation steps (effective batch 128) run_training_lavis.py run_training_lavis.py, and stops after more than 1000 optimizer steps run_training_lavis.py. These do not match.
• Model selection. The paper says early stopping on validation app. B. The code evaluates every 100 steps and saves a checkpoint each time run_training_lavis.py, but does not choose one.
• Training target. The code trains on the first answer only run_training_lavis.py. The paper does not say how multiple answers are used.
• Trainable part. The code freezes everything except the Q-Former run_training_lavis.py, consistent with §4.1.
• Length penalty and prompt. Length penalty -1 run_training_lavis.py and prompt "Question: {} Short answer:" run_training_lavis.py agree with app. B.
• Beam size. The paper's table gives beam size 5 app. B. The inference scripts do not pass a beam count, so 5 applies only if LAVIS's default is 5 (not checked).
• Zero-shot numbers. The README example prints val scores of 8.06 (instruct) and 12.51 (pretrain) [README]. The paper's zero-shot Wikidata values are 7.4 (InstructBLIP) and 11.3 (BLIP2) fig. 3. The split of fig. 3 is not stated, so this may be a split difference, but it is unresolved.
• Image preprocessing. Images are resized to 224×224 without keeping aspect ratio run_training_lavis.py run_training_lavis.py. The paper gives only the size.

Pitfalls

• Metric mismatch. The paper's written metric (VQA accuracy, one-year tolerance) is not what the released code computes. Implement the code if you want to reproduce published scores; implement the paper if you want the stated metric. The difference is not measured here.
• Harmonic mean vs pooled mean. The overall is over the two splits, not over all questions. Pooled averages will not match Overall.
• Parameter counts disagree. FiD is "T5-large (660M)" in the text §4.2, "0.4B" in app. B, and "1B" for CLIP->FiD in app. C (which may include CLIP). BLIP-2 is 12B in the table and 12.1B in app. B.
• PaLM entity count. The text feeds five entities to the reader §4.2. The appendix says PaLM uses only the top-1 predicted entity app. B. Follow the appendix if reproducing PaLM.
• Weighted cosine. The text says the top-5 entities are chosen by "weighted cosine similarity" §4.2. The weights are not stated, and no CLIP code is released.
• Split naming. "Unseen Entity" and "Unseen Question" are reported on both parts of the dataset. The main table does not state its split; the validation numbers are in app. C.
• Dataset availability and license. The paper footnote points to a project page. The project page and the code repo do not state a license for the dataset. The data draws on Wikipedia, Wikidata and nine image datasets §3 app. A.4; the paper states no license terms for them. Check these before redistributing the data or derived files.
• Costs. Training runs use 6 to 48 hours on 8 to 64 accelerators app. B. Building the Wikidata part uses 300 relations with one or two human-written templates each §3.2. Entity recognition needs a 100K-entry Wikipedia KB §4.2.
• Annotation checks are small. The 95% human-answer accuracy comes from 200 expert-checked examples. The 97% Wikidata answer-in-article rate comes from 200 sampled pairs app. A.1 app. A.2.
• Split design. The splits are meant to make memorizing the training set useless §1 §3.3. Entities in Unseen Entity are held out from training §3.3.
• Version. This card follows arXiv v5 (updated 2023-10-17) as in the local LaTeX source. Section and table numbers refer to that version.

Lineage

• Builds on: the OVEN visual entity recognition data and CLIP2CLIP entity matching, used for the With-KB pipeline §3 §4.2. Prior benchmarks it argues against: OK-VQA, ViQuAE and K-VQA §2. Image sources: nine datasets used in OVEN §3.
• Compared against: PaLI-17B, PaLI-X, BLIP-2, InstructBLIP, PaLM 5-shot question-only, FiD, OFA-large and mPLUG-owl §4 app. B tab. 4.
• Superseded by: not known from content in this wiki.