--- kind: concept slug: visual-question-answering-evaluation title: Visual question-answering evaluation summary: >- Scoring a vision-language model by asking it questions with known answers. Three uses: checking generated images against prompts (TIFA; Spearman 59.7 for mPLUG vs 33.2 for CLIPScore), benchmarking video models (VSI-Bench; best model 48.8 vs humans 79.2) and answering questions about 3D scans (ScanQA; EM@1 23.45 vs 22.31 for a 2D baseline). Costs one VQA call per question. tags: [evaluation, visual-question-answering, text-to-image, faithfulness, video, spatial-reasoning, 3d-scene-understanding, benchmark] papers: ["2303.11897", "2412.14171", "2112.10482"] related: [text-to-image-generation, human-evaluation, image-captioning, multimodal-language-model, benchmark-validity, chain-of-thought-prompting, self-consistency, visual-grounding] status: draft updated_at: 2026-10-24 --- ## Overview Score a model's output by asking a visual question-answering (VQA) model questions whose answers are known, then counting how many it gets right. The method is used in three ways on this page: - **Checking a generated image against its prompt.** The prompt is split into small facts, each fact becomes a question with a known answer, and a VQA model answers those questions about the image. The score is the fraction answered as the prompt dictates [arxiv:2303.11897 §3] [eq. 1]. - **Benchmarking a video model.** A fixed set of questions about each video is answered by the model, and results are reported per task: accuracy for multiple-choice questions and mean relative accuracy for numeric ones [arxiv:2412.14171 §3] [§4.1]. - **Answering questions about a 3D indoor scan.** ScanQA asks free-form questions about a room scan and scores the answers by exact match and by captioning metrics. One split also scores object localization [arxiv:2112.10482 §3.1] [§5.1]. The three uses rely on the same idea but produce different numbers. TIFA's score is a fraction of questions answered correctly per image; VSI-Bench reports accuracy per task on a benchmark; ScanQA reports exact-match and captioning scores on its own test splits. Compare scores only within one use. All numbers below are author-reported on each paper's own benchmark. ## Variants ### Image faithfulness: question writing and filtering The TIFA pipeline has three replaceable parts: the question writer, the question filter and the VQA answerer. Each is a choice. #### Full pipeline with a large language model as question writer The paper's setup: one in-context LM call per prompt writes two questions per element, one yes/no and one multiple-choice, using 15 hand-annotated examples [arxiv:2303.11897 §3.1]. A QA model (UnifiedQA) keeps a tuple only if its multiple-choice and free-form answers agree with the generated answer [§3.2]. Card: [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](/papers/2303.11897). #### Pre-built question set The authors released TIFA v1.0: 4,081 prompts with 25,829 questions, all generated with GPT-3 and pre-filtered [arxiv:2303.11897 §4] [tab. 1]. You skip the question writer and run only the VQA model [§4] [app. D]. Expect the questions to be fixed to that set, so results compare only with other scores computed on it. #### Open question writer For reproducibility the authors fine-tuned LLaMA 2 (7B) on the TIFA v1.0 tuples to write questions from captions [arxiv:2303.11897 §4.3]. The card does not record how closely its questions match the GPT-3 tuples; check §4.3 before relying on it. #### VQA answering model for images Five general vision-language models were compared: ViLT, OFA, GIT, BLIP-2 and mPLUG [arxiv:2303.11897 §3.3]. Four produce free text, which is matched to the closest choice; BLIP-2 receives the choices in its prompt [§3.3]. The paper recommends mPLUG-large [§3.3] [§5.4]. ### Video benchmark: fixed questions, per-task scores VSI-Bench is a benchmark, not a per-image score. It has over 5,000 question- answer pairs from 288 indoor videos drawn from three public 3D scan datasets [arxiv:2412.14171 §3.1] [app. B.1]. Eight tasks cover object counts, absolute distance, object size, room size, relative distance, relative direction, route plan and appearance order [§3.2] [fig. 3]. Multiple-choice tasks are scored by accuracy. Numeric tasks use mean relative accuracy: a prediction counts as correct at each of ten thresholds from 0.50 to 0.95 when its relative error is below one minus the threshold, and the result is averaged over thresholds [§4.1] [eq. 1]. Models are evaluated zero-shot with greedy decoding [app. C.1]. Card: [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](/papers/2412.14171). ### 3D scene question answering: free-form answers and localization ScanQA uses 41,363 questions over 800 ScanNet scenes, with one test split that has object annotations and one that does not [arxiv:2112.10482 §3.3]. A model gives a free-form answer and, on the split with objects, the boxes of the objects the question refers to [§3.1] [app. B]. Answers are scored by exact match at one and ten candidates (EM@1, EM@10) and by captioning metrics such as BLEU, ROUGE, METEOR, CIDEr and SPICE [§5.1]. The authors add the captioning metrics because answers to location questions vary in wording [§3.3]. Card: [ScanQA: 3D Question Answering for Spatial Scene Understanding](/papers/2112.10482). The paper's ablation removes heads from the joint model. Dropping the localization head gives EM@1 20.46 and dropping the object-class head gives 18.31, against 23.45 with all three heads, on the split with objects [arxiv:2112.10482 tab. 4]. Training answering and localization together is the design choice this evaluation rests on. ### Answering through an intermediate map For relative-distance questions, the paper asks the model first to place each object on a 10 x 10 grid, then answers from that map [arxiv:2412.14171 §6.2] [app. B.4]. On Gemini-1.5 Pro this raises relative-distance accuracy from 46.0 to 56.0 [tab. 3]. Ground-truth maps reach 66.0, so the model-made map recovers only part of the gap; treat the ground-truth row as an upper bound, not a method [§6.2]. A 20 x 20 grid does worse with the model's own map (54.0) [tab. 3]. ## When to use which Use the table to choose a use and its settings. Figures are the papers' own. | Situation | Use | Why | |---|---|---| | Score a generated image with no reference image | Pre-built TIFA set with mPLUG | mPLUG has the highest human-rating correlation in TIFA, 59.7 Spearman [arxiv:2303.11897 tab. 2]; 76.8 correlation with human VQA answers [tab. 3]. About 0.25 s per question [§5.4]. | | Same image score, lower latency needed | ViLT | 0.08 s per question, but 49.3 Spearman with human ratings [§5.4] [tab. 2]. | | Same image score, middle ground on speed and agreement | GIT or OFA | About 0.25 s per question with 54.5 and 49.6 Spearman [§5.4] [tab. 2]. | | Need a score for a whole image, not per element | Captions or CLIPScore | Caption metrics and CLIPScore correlate at 18.3 to 34.0 [tab. 2], well below the VQA variants. | | Need to know which image element failed | TIFA per-element scores | Scores split by element category, e.g. counting or spatial relations [tab. 4]. | | Compare video models on spatial questions | VSI-Bench, per-task scores | Report each task separately; the tasks differ widely [arxiv:2412.14171 tab. 1]. | | Numeric video answers (counts, sizes, distances) | Mean relative accuracy | Exact match is too strict for estimates; MRA gives partial credit [§4.1] [eq. 1]. | | Relative-distance errors on video | Ask for a 10 x 10 map first | 46.0 to 56.0 on Gemini-1.5 Pro [tab. 3]. Check the model's map quality first, since accuracy falls with distance [app. B.5]. | | Expecting chain-of-thought to help spatial questions | Test it; do not assume | On VSI-Bench, zero-shot CoT and tree-of-thoughts lower the average [§5.2] [fig. 8]. The same model gains on VideoMME with CoT [tab. 2]. | | Free-form answers about a 3D scan | EM@1 and EM@10, with captioning metrics | Exact match alone punishes different wording. The paper reports both families [arxiv:2112.10482 §3.3] [§5.1]. | | Object grounding is part of the task | Acc@0.25 and Acc@0.5 on the split with objects | The split without objects has no object annotations, so localization cannot be scored there [arxiv:2112.10482 §3.3] [app. B]. | | Check whether 3D input is worth its cost | A 2D-frame baseline in the same table | The best 2D baseline is 1.1 EM@1 points below ScanQA on the split with objects [arxiv:2112.10482 tab. 3]. | ### Pitfalls - **Cost.** One VQA call per question per image. With the TIFA paper's mean of 6.3 questions per prompt and mPLUG at about 0.25 s per question on an A40, one image takes about 1.6 s [arxiv:2303.11897 §5.4] [§6] [tab. 1]. The authors call this too slow for filtering large datasets [§6]. - **Filter threshold.** TIFA keeps a tuple when word F1 with the answer is above 0.7 [§3.2]. The released code uses 0.6 (see the [paper card](/papers/2303.11897)). Pick one and record it. - **Regenerated questions are not the benchmark.** Questions written by another LM differ in wording and count, so their scores are not comparable with TIFA v1.0 [arxiv:2303.11897 §4.3] [app. D]. - **Human scores are not an upper bound for images.** Human VQA answers include a "none of the above" option that models do not get [arxiv:2303.11897 §5.4] [app. C]. - **Human scores are not matched for video either.** Humans had unlimited time and could replay the clip; the authors argue that models also reprocess their input [arxiv:2412.14171 app. C.2]. The number of human evaluators is not stated. - **Ground truth does not score 100%.** Real COCO images also fall short, from caption noise and VQA errors [arxiv:2303.11897 §5.3]. Do not treat 100% as a target. - **Assumption about the VQA model.** The method assumes the answering model is good enough; the TIFA authors flag anime and abstract art as domains where that may fail [arxiv:2303.11897 §6]. - **Long prompts.** Average image scores fall when a prompt names more than five entities, while real images do not show the same drop [arxiv:2303.11897 §5.3] [fig. 9]. - **Input regimes differ by model.** On VSI-Bench, Gemini gets the whole video, open-source models get 32 uniformly sampled frames, and GPT-4o gets 16 [arxiv:2412.14171 app. C.3]. Table 1 places them side by side, so treat cross-model gaps with that in mind. - **Small subsets are noisy.** The 400-question VSI-Bench subset has 50 questions per task and no confidence intervals are reported [arxiv:2412.14171 §4.1] [tab. 1]. Differences of a few points are hard to read. - **Blind runs are a sanity check.** The VSI-Bench authors also ran evaluations with the video withheld; six models average below the frequency baseline there [arxiv:2412.14171 app. C.4]. Object size scores well above chance without video, which the authors attribute to knowledge from language-model training [app. C.4]. - **Zero ground truth breaks MRA.** MRA divides by the ground truth, and the paper does not say what happens when it is zero [arxiv:2412.14171 eq. 1]. - **The 3D human reference is weak.** The best of the MTurk workers reached EM@1 51.6, on a sample of test questions only [arxiv:2112.10482 §5.2]. Treat it as a rough ceiling, not a benchmark. - **Answers come from a closed set.** The official code predicts only answers seen in training, with no cap on their number. Answers outside that set cannot be produced [see the paper card](/papers/2112.10482). - **Localization uses one object in the code.** The official code takes only the first annotated object as the localization target. Questions about several objects are handled only by an appendix variant [see the paper card](/papers/2112.10482) [arxiv:2112.10482 app. A]. - **Pick settings before reading test numbers.** The appendix sweep reports settings that score higher on the test split, such as hidden size 128 with EM@1 24.38 against 23.45 for the main setting [arxiv:2112.10482 app. B]. The main setting is hidden size 256 with two layers [§4]. Do not report the best sweep value as the main result. - **The text and the code differ on training targets.** The paper describes 0/1 answer targets; the code uses graded soft targets [see the paper card](/papers/2112.10482). A reimplementation from the text trains differently. - **Data terms are not checked here.** The scenes come from ScanNet and the questions from ScanRefer captions. The terms of use of both were not checked, so check them before reusing the data [arxiv:2112.10482 §3.2]. ## Implementation sketch The image pipeline writes and filters questions once per prompt, then answers them once per image. The VQA step matches the procedure in [arxiv:2303.11897 §3]. ```pseudocode # Write and filter questions once per prompt [arxiv:2303.11897 §3.1] [§3.2] # Input: prompt, lm (question writer), qa_model (filter) # Output: list of (question, choices, answer) tuples def build_questions(prompt, lm, qa_model): # One LM call writes tuples from the prompt's elements [§3.1] candidates = lm.write_questions(prompt) kept = [] for question, choices, answer in candidates: # Keep a tuple only if the QA model agrees on both checks [§3.2] mc_ok = qa_model.multiple_choice(prompt, question, choices) == answer free = qa_model.free_form(prompt, question) if mc_ok and word_f1(free, answer) > 0.7: kept.append((question, choices, answer)) return kept # Score one image against its prompt's questions [arxiv:2303.11897 §3.3] # Input: questions from build_questions, image, vqa_model # Output: faithfulness in [0, 1] def score(questions, image, vqa_model): hits = [] for question, choices, answer in questions: # Free-form answer, then the closest listed choice [§3.3] free = vqa_model.answer(image, question, choices) predicted = closest_choice(free, choices) hits.append(predicted == answer) return sum(hits) / len(hits) ``` For numeric video answers, the benchmark's scoring step is a separate function. It follows [arxiv:2412.14171 §4.1] and [eq. 1]. ```pseudocode # Mean relative accuracy for one numeric answer [arxiv:2412.14171 §4.1] [eq. 1] # prediction, target: floats; target must be non-zero (see Pitfalls) def mean_relative_accuracy(prediction, target): # Relative error against the ground truth [eq. 1] relative_error = abs(prediction - target) / target # Ten thresholds from 0.50 to 0.95 [§4.1] thresholds = [0.50 + 0.05 * i for i in range(10)] # One pass/fail per threshold; the paper uses a strict "<" [eq. 1] passed = [relative_error < 1 - t for t in thresholds] # Average over thresholds [§4.1] return sum(passed) / len(thresholds) ``` For free-form 3D answers, the scoring step is exact match at K. The paper's text normalisation for string comparison is not stated. ```pseudocode # Exact match at K for one question [arxiv:2112.10482 §5.1] # Input: ranked answer strings, ground-truth answer strings, k # Output: 1 if a ground-truth answer is in the top k, else 0 def exact_match_at_k(ranked, ground_truths, k): # Compare each of the top k predictions with the ground-truth set top = ranked[:k] return int(any(answer in ground_truths for answer in top)) ``` ## Contradictions Within the papers, the numbers disagree in several places: - TIFA's filter threshold: 0.7 in the paper text [arxiv:2303.11897 §3.2], 0.6 in the released code (see the [paper card](/papers/2303.11897)). - VSI-Bench, chain-of-thought on VideoMME: the text reports a 1.6-point gain [arxiv:2412.14171 §5.2], while the table gives 77.2 and 79.8, a 2.6-point gain [tab. 2]. - VSI-Bench, self-consistency temperature: 1.0 in the main text [§5.2], 0.7 in the appendix [app. B.3]. - ScanQA, BLEU on the split without objects: the table prints BLEU-3 as 16.53 for ScanRefer+MCAN (pipeline), above its BLEU-2 (10.98) and BLEU-4 (0.76) values [arxiv:2112.10482 tab. 3]. Check the source before using these three values. Across the two uses, the paper results point in different directions for chain-of-thought: it helps Gemini-1.5 Pro on VideoMME [tab. 2] and lowers the average on VSI-Bench [§5.2] [fig. 8]. These are different benchmarks, so this is not a direct contradiction, but one benchmark's result does not predict the other's. The two uses report different quantities. Correlation with human ratings [arxiv:2303.11897 tab. 2], agreement with human VQA answers [tab. 3] and per-task accuracy or MRA [arxiv:2412.14171 tab. 1] are not interchangeable. ## Papers - [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](/papers/2303.11897): question-generation, filtering and VQA scoring pipeline with a released 4,081-prompt benchmark. - [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](/papers/2412.14171): VSI-Bench, a video spatial benchmark scored by accuracy and MRA, with probes on prompting and cognitive maps. - [ScanQA: 3D Question Answering for Spatial Scene Understanding](/papers/2112.10482): free-form questions about 3D indoor scans, scored by EM@K and captioning metrics, with object localization on one split.