Visual question-answering evaluation

Overview

Score a model's output by asking a visual question-answering (VQA) model questions whose answers are known, then counting how many it gets right. The method is used in three ways on this page:

• Checking a generated image against its prompt. The prompt is split into small facts, each fact becomes a question with a known answer, and a VQA model answers those questions about the image. The score is the fraction answered as the prompt dictates 2303.11897 §3 eq. 1.
• Benchmarking a video model. A fixed set of questions about each video is answered by the model, and results are reported per task: accuracy for multiple-choice questions and mean relative accuracy for numeric ones 2412.14171 §3 §4.1.
• Answering questions about a 3D indoor scan. ScanQA asks free-form questions about a room scan and scores the answers by exact match and by captioning metrics. One split also scores object localization 2112.10482 §3.1 §5.1.

The three uses rely on the same idea but produce different numbers. TIFA's score is a fraction of questions answered correctly per image; VSI-Bench reports accuracy per task on a benchmark; ScanQA reports exact-match and captioning scores on its own test splits. Compare scores only within one use. All numbers below are author-reported on each paper's own benchmark.

Variants

Image faithfulness: question writing and filtering

The TIFA pipeline has three replaceable parts: the question writer, the question filter and the VQA answerer. Each is a choice.

Full pipeline with a large language model as question writer

The paper's setup: one in-context LM call per prompt writes two questions per element, one yes/no and one multiple-choice, using 15 hand-annotated examples 2303.11897 §3.1. A QA model (UnifiedQA) keeps a tuple only if its multiple-choice and free-form answers agree with the generated answer §3.2. Card: TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering.

Pre-built question set

The authors released TIFA v1.0: 4,081 prompts with 25,829 questions, all generated with GPT-3 and pre-filtered 2303.11897 §4 tab. 1. You skip the question writer and run only the VQA model §4 app. D. Expect the questions to be fixed to that set, so results compare only with other scores computed on it.

Open question writer

For reproducibility the authors fine-tuned LLaMA 2 (7B) on the TIFA v1.0 tuples to write questions from captions 2303.11897 §4.3. The card does not record how closely its questions match the GPT-3 tuples; check §4.3 before relying on it.

VQA answering model for images

Five general vision-language models were compared: ViLT, OFA, GIT, BLIP-2 and mPLUG 2303.11897 §3.3. Four produce free text, which is matched to the closest choice; BLIP-2 receives the choices in its prompt §3.3. The paper recommends mPLUG-large §3.3 §5.4.

Video benchmark: fixed questions, per-task scores

VSI-Bench is a benchmark, not a per-image score. It has over 5,000 question- answer pairs from 288 indoor videos drawn from three public 3D scan datasets 2412.14171 §3.1 app. B.1. Eight tasks cover object counts, absolute distance, object size, room size, relative distance, relative direction, route plan and appearance order §3.2 fig. 3.

Multiple-choice tasks are scored by accuracy. Numeric tasks use mean relative accuracy: a prediction counts as correct at each of ten thresholds from 0.50 to 0.95 when its relative error is below one minus the threshold, and the result is averaged over thresholds §4.1 eq. 1. Models are evaluated zero-shot with greedy decoding app. C.1. Card: Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces.

3D scene question answering: free-form answers and localization

ScanQA uses 41,363 questions over 800 ScanNet scenes, with one test split that has object annotations and one that does not 2112.10482 §3.3. A model gives a free-form answer and, on the split with objects, the boxes of the objects the question refers to §3.1 app. B. Answers are scored by exact match at one and ten candidates (EM@1, EM@10) and by captioning metrics such as BLEU, ROUGE, METEOR, CIDEr and SPICE §5.1. The authors add the captioning metrics because answers to location questions vary in wording §3.3. Card: ScanQA: 3D Question Answering for Spatial Scene Understanding.

The paper's ablation removes heads from the joint model. Dropping the localization head gives EM@1 20.46 and dropping the object-class head gives 18.31, against 23.45 with all three heads, on the split with objects 2112.10482 tab. 4. Training answering and localization together is the design choice this evaluation rests on.

Answering through an intermediate map

For relative-distance questions, the paper asks the model first to place each object on a 10 x 10 grid, then answers from that map 2412.14171 §6.2 app. B.4. On Gemini-1.5 Pro this raises relative-distance accuracy from 46.0 to 56.0 tab. 3. Ground-truth maps reach 66.0, so the model-made map recovers only part of the gap; treat the ground-truth row as an upper bound, not a method §6.2. A 20 x 20 grid does worse with the model's own map (54.0) tab. 3.

When to use which

Use the table to choose a use and its settings. Figures are the papers' own.

Pitfalls

• Cost. One VQA call per question per image. With the TIFA paper's mean of 6.3 questions per prompt and mPLUG at about 0.25 s per question on an A40, one image takes about 1.6 s 2303.11897 §5.4 §6 tab. 1. The authors call this too slow for filtering large datasets §6.
• Filter threshold. TIFA keeps a tuple when word F1 with the answer is above 0.7 §3.2. The released code uses 0.6 (see the paper card). Pick one and record it.
• Regenerated questions are not the benchmark. Questions written by another LM differ in wording and count, so their scores are not comparable with TIFA v1.0 2303.11897 §4.3 app. D.
• Human scores are not an upper bound for images. Human VQA answers include a "none of the above" option that models do not get 2303.11897 §5.4 app. C.
• Human scores are not matched for video either. Humans had unlimited time and could replay the clip; the authors argue that models also reprocess their input 2412.14171 app. C.2. The number of human evaluators is not stated.
• Ground truth does not score 100%. Real COCO images also fall short, from caption noise and VQA errors 2303.11897 §5.3. Do not treat 100% as a target.
• Assumption about the VQA model. The method assumes the answering model is good enough; the TIFA authors flag anime and abstract art as domains where that may fail 2303.11897 §6.
• Long prompts. Average image scores fall when a prompt names more than five entities, while real images do not show the same drop 2303.11897 §5.3 fig. 9.
• Input regimes differ by model. On VSI-Bench, Gemini gets the whole video, open-source models get 32 uniformly sampled frames, and GPT-4o gets 16 2412.14171 app. C.3. Table 1 places them side by side, so treat cross-model gaps with that in mind.
• Small subsets are noisy. The 400-question VSI-Bench subset has 50 questions per task and no confidence intervals are reported 2412.14171 §4.1 tab. 1. Differences of a few points are hard to read.
• Blind runs are a sanity check. The VSI-Bench authors also ran evaluations with the video withheld; six models average below the frequency baseline there 2412.14171 app. C.4. Object size scores well above chance without video, which the authors attribute to knowledge from language-model training app. C.4.
• Zero ground truth breaks MRA. MRA divides by the ground truth, and the paper does not say what happens when it is zero 2412.14171 eq. 1.
• The 3D human reference is weak. The best of the MTurk workers reached EM@1 51.6, on a sample of test questions only 2112.10482 §5.2. Treat it as a rough ceiling, not a benchmark.
• Answers come from a closed set. The official code predicts only answers seen in training, with no cap on their number. Answers outside that set cannot be produced see the paper card.
• Localization uses one object in the code. The official code takes only the first annotated object as the localization target. Questions about several objects are handled only by an appendix variant see the paper card 2112.10482 app. A.
• Pick settings before reading test numbers. The appendix sweep reports settings that score higher on the test split, such as hidden size 128 with EM@1 24.38 against 23.45 for the main setting 2112.10482 app. B. The main setting is hidden size 256 with two layers §4. Do not report the best sweep value as the main result.
• The text and the code differ on training targets. The paper describes 0/1 answer targets; the code uses graded soft targets see the paper card. A reimplementation from the text trains differently.
• Data terms are not checked here. The scenes come from ScanNet and the questions from ScanRefer captions. The terms of use of both were not checked, so check them before reusing the data 2112.10482 §3.2.

Implementation sketch

The image pipeline writes and filters questions once per prompt, then answers them once per image. The VQA step matches the procedure in 2303.11897 §3.

# Write and filter questions once per prompt  [arxiv:2303.11897 §3.1] [§3.2]
# Input:  prompt, lm (question writer), qa_model (filter)
# Output: list of (question, choices, answer) tuples

def build_questions(prompt, lm, qa_model):
    # One LM call writes tuples from the prompt's elements  [§3.1]
    candidates = lm.write_questions(prompt)
    kept = []
    for question, choices, answer in candidates:
        # Keep a tuple only if the QA model agrees on both checks  [§3.2]
        mc_ok = qa_model.multiple_choice(prompt, question, choices) == answer
        free = qa_model.free_form(prompt, question)
        if mc_ok and word_f1(free, answer) > 0.7:
            kept.append((question, choices, answer))
    return kept


# Score one image against its prompt's questions  [arxiv:2303.11897 §3.3]
# Input:  questions from build_questions, image, vqa_model
# Output: faithfulness in [0, 1]

def score(questions, image, vqa_model):
    hits = []
    for question, choices, answer in questions:
        # Free-form answer, then the closest listed choice  [§3.3]
        free = vqa_model.answer(image, question, choices)
        predicted = closest_choice(free, choices)
        hits.append(predicted == answer)
    return sum(hits) / len(hits)

For numeric video answers, the benchmark's scoring step is a separate function. It follows 2412.14171 §4.1 and eq. 1.

# Mean relative accuracy for one numeric answer  [arxiv:2412.14171 §4.1] [eq. 1]
# prediction, target: floats; target must be non-zero (see Pitfalls)

def mean_relative_accuracy(prediction, target):
    # Relative error against the ground truth  [eq. 1]
    relative_error = abs(prediction - target) / target
    # Ten thresholds from 0.50 to 0.95  [§4.1]
    thresholds = [0.50 + 0.05 * i for i in range(10)]
    # One pass/fail per threshold; the paper uses a strict "<"  [eq. 1]
    passed = [relative_error < 1 - t for t in thresholds]
    # Average over thresholds  [§4.1]
    return sum(passed) / len(thresholds)

For free-form 3D answers, the scoring step is exact match at K. The paper's text normalisation for string comparison is not stated.

# Exact match at K for one question  [arxiv:2112.10482 §5.1]
# Input:  ranked answer strings, ground-truth answer strings, k
# Output: 1 if a ground-truth answer is in the top k, else 0

def exact_match_at_k(ranked, ground_truths, k):
    # Compare each of the top k predictions with the ground-truth set
    top = ranked[:k]
    return int(any(answer in ground_truths for answer in top))

Contradictions

Within the papers, the numbers disagree in several places:

• TIFA's filter threshold: 0.7 in the paper text 2303.11897 §3.2, 0.6 in the released code (see the paper card).
• VSI-Bench, chain-of-thought on VideoMME: the text reports a 1.6-point gain 2412.14171 §5.2, while the table gives 77.2 and 79.8, a 2.6-point gain tab. 2.
• VSI-Bench, self-consistency temperature: 1.0 in the main text §5.2, 0.7 in the appendix app. B.3.
• ScanQA, BLEU on the split without objects: the table prints BLEU-3 as 16.53 for ScanRefer+MCAN (pipeline), above its BLEU-2 (10.98) and BLEU-4 (0.76) values 2112.10482 tab. 3. Check the source before using these three values.

Across the two uses, the paper results point in different directions for chain-of-thought: it helps Gemini-1.5 Pro on VideoMME tab. 2 and lowers the average on VSI-Bench §5.2 fig. 8. These are different benchmarks, so this is not a direct contradiction, but one benchmark's result does not predict the other's.

The two uses report different quantities. Correlation with human ratings 2303.11897 tab. 2, agreement with human VQA answers tab. 3 and per-task accuracy or MRA 2412.14171 tab. 1 are not interchangeable.

Papers

• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering: question-generation, filtering and VQA scoring pipeline with a released 4,081-prompt benchmark.
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces: VSI-Bench, a video spatial benchmark scored by accuracy and MRA, with probes on prompting and cognitive maps.
• ScanQA: 3D Question Answering for Spatial Scene Understanding: free-form questions about 3D indoor scans, scored by EM@K and captioning metrics, with object localization on one split.