---
kind: concept
slug: visual-question-answering
title: Visual question answering
summary: >-
  Answering a text question about an image or video. Three strands in the cards: scene-text VQA (PixelM4C, 41.23% TextVQA val without ST-VQA data, author-reported); video QA (survey-tabulated scores, not like-for-like); and dialog-based video QA (AVSD, where dialog history gives the largest gain). Three cards so far.
tags: [vqa, visual-question-answering, text-vqa, video-qa, dialog, multimodal, survey]
papers: ["1901.09107", "2105.05486", "2406.05615"]
related: [visual-question-answering-evaluation, conversational-question-answering, optical-character-recognition, multimodal-language-model, vision-language-pretraining, multimodal-fusion, hard-negative-mining, image-captioning]
status: draft
updated_at: 2026-11-06
---

## Overview

Visual question answering (VQA) gives a model an image or a video and a
question in text, and asks for an answer in text. The answer is either picked
from a fixed vocabulary, picked from a list of candidates, or generated as free
text. Three families appear in the current cards:

- **Scene-text VQA**, where the question concerns words written in the scene
  (signs, labels, posters). The answer model needs an OCR component to read
  those words [arxiv:2105.05486 §1].
- **Video question answering**, where the model must reason over frames in
  order, not over a single picture [arxiv:2406.05615 §2].
- **Dialog-based video QA**, where the question is one turn of a conversation
  about a video, so earlier turns are part of the input [arxiv:1901.09107 §1].

The evidence base is still thin: three cards list this slug. For generic VQA
evaluation, see [Visual question-answering evaluation](/concepts/visual-question-answering-evaluation).

## Variants

### Scene-text VQA: PixelM4C

The answer model is M4C, an iterative answer-prediction transformer for
TextVQA [arxiv:2105.05486 §5.2]. PixelM4C keeps M4C and changes its text
source. It runs a TextOCR-trained Mask TextSpotter v3 as a frozen module and
passes the spotter's recognised tokens, with features taken from the
decoder's last hidden state at the end-of-sequence token, into M4C
[arxiv:2105.05486 §5.2] [tab. 6]. Card: [TextOCR/PixelM4C card](/papers/2105.05486).

Reported TextVQA val accuracy, author-reported [arxiv:2105.05486 tab. 6]:
- without ST-VQA data: PixelM4C 41.23 against M4C 39.40;
- with ST-VQA data: PixelM4C 42.12 against M4C 40.55.

The same paper's TextCaps variant has a lower CIDEr than its baseline (85.32
against 89.60), and the authors report no significant gain there except on
BLEU-4 [arxiv:2105.05486 §5.2] [tab. 6].

### Dialog-based video QA: AVSD baseline

The AVSD paper defines a benchmark of 11,816 dialogs about Charades videos, each
with ten question-answer rounds and a closing summary [arxiv:1901.09107 §3.1].
Its baseline ranks answers instead of generating them. The authors chose this
because discriminative ranking worked better than generation in visual dialog
[arxiv:1901.09107 §4 Decoder Model]. The model works as follows:

- an LSTM encodes the script and the earlier rounds, and a second LSTM encodes
  the current question;
- the video and the audio each enter as one fixed-size vector, from I3D and
  AENet respectively [arxiv:1901.09107 §4 Implementation Details];
- the four vectors are concatenated and passed through one tanh layer;
- each of 100 candidate answers is encoded and scored by an inner product with
  the fused vector, and a softmax over the candidates gives the training loss
  [arxiv:1901.09107 eq. 1].

Card: [Audio-Visual Scene-Aware Dialog](/papers/1901.09107).

Reported test MRR, author-reported, single runs [arxiv:1901.09107 tab. 3]:
- question only: 36.12;
- question plus dialog history: 50.40;
- question, dialog history and video: 53.41;
- the same with audio added: 53.03.

Shuffling the order of the earlier rounds lowers MRR for each row, but the
model was tested on shuffled dialogs without being retrained on them
[arxiv:1901.09107 tab. 4]. For the audio-related questions, mean rank is 6.91
with the question alone and 6.69 when audio is added [arxiv:1901.09107 tab. 5].

### Video question answering

The survey splits the methods by architecture [arxiv:2406.05615 §4]:

- **Pre-Transformer** models use CNN or RNN encoders and fusion by attention,
  multiplication or graphs. The best listed MSRVTT accuracy in this family is
  40.0 (QueST) [arxiv:2406.05615 tab. 3].
- **Transformer-based** models come in shared, stacked and dual forms. The
  best listed MSRVTT accuracy is 45.0 (LAVENDER, stacked) [arxiv:2406.05615
  tab. 3].
- **LLM-augmented** models pass visual tokens, produced by a semantic
  translator (a Q-Former with learnable queries or a linear projection), into
  an LLM that generates the answer [arxiv:2406.05615 §4.3]. The best listed
  MSRVTT accuracy is 58.9 (LLaMA-Vid, EVA-G with Vicuna) [arxiv:2406.05615
  tab. 3].

All these scores are copied from the cited papers. The survey does not say
whether they were re-run, and the backbones differ [arxiv:2406.05615 tab. 3].
Card: [video-language survey card](/papers/2406.05615).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Answers depend on text in the image | Scene-text VQA with an OCR module | The text-VQA gain comes from a better spotter and from feeding its tokens to the answer model [arxiv:2105.05486 §5.2] [tab. 5] |
| OCR module cannot be trained on text-dense data | Stay with the baseline reader | PixelM4C's gain depends on a TextOCR-trained spotter; the cards give no result for other readers [arxiv:2105.05486 tab. 6] |
| Questions are follow-ups in a conversation about a video | Dialog-based model with the history in the question encoder | In AVSD, adding dialog history raised test MRR from 36.12 to 50.40 [arxiv:1901.09107 tab. 3] |
| You can score a fixed candidate list | Discriminative ranking | The AVSD authors report it outperforming generation in visual dialog [arxiv:1901.09107 §4 Decoder Model]; the cards give no generative comparison on AVSD |
| Audio is available and the main score matters | Test with and without audio on your own data | In AVSD, audio did not raise MRR on top of video (53.41 against 53.03) [arxiv:1901.09107 tab. 3] |
| Question concerns events over time | Video QA model | The survey treats temporal modelling as a core challenge for video-language tasks [arxiv:2406.05615 §3] |
| Large compute and a pretrained LLM are available | LLM-augmented video QA | The survey reports the highest VideoQA score for this family [arxiv:2406.05615 tab. 3], but the rows differ in backbone size, so the gap is not a like-for-like result [arxiv:2406.05615 tab. 3] |
| Small models or limited compute | Transformer-based or pre-Transformer video QA | The survey notes adapters or prompt vectors as options when compute or data is limited [arxiv:2406.05615 §5.2] |

The cards give no measured latency, memory or data-efficiency numbers for
VQA. Pick a design from the constraints above and check its cost in your own
setting.

## Implementation sketch

The common core: encode the image (or frames), encode the question, combine
the two, and produce an answer. Where variants differ, the notes say how.

```pseudocode
# Generic VQA step  [arxiv:2406.05615 §4.3] [arxiv:2105.05486 §5.2]
#                   [arxiv:1901.09107 §4]
# Input:  image or frames; question text; optional dialog history
# Output: answer text

def answer(visual_input, question, history=None):
    # 1. Encode visual input (CNN, ViT, frozen detector, or fixed-size I3D vector)
    visual_tokens = visual_encoder(visual_input)
    # 2. Optional: read text in the scene; PixelM4C keeps tokens with
    #    recognition score >= 0.2 and caps them at 200  [arxiv:2105.05486 §5.2]
    ocr_tokens = scene_text_reader(visual_input)   # frozen in PixelM4C
    # 3. Optional: encode earlier dialog rounds  [arxiv:1901.09107 §4]
    history_state = encode_history(history) if history else None
    # 4. Fuse question, visual tokens and OCR tokens
    #    Fusion can be an iterative transformer (M4C) or an LLM prompt
    #    [arxiv:2406.05615 §4.3]; AVSD concatenates and applies one tanh layer
    fused = fuse(question, visual_tokens, ocr_tokens, history_state)
    # 5. Predict: score a fixed vocabulary, rank a candidate list, or
    #    generate free text  [arxiv:1901.09107 §4 Decoder Model]
    return decode_answer(fused)
```

## Contradictions

- **Video QA ranking.** The survey says LLM-augmented models have begun to
  outperform Transformer-based models on VideoQA [arxiv:2406.05615 §4.4].
  The table rows support that direction but use different backbones and
  parameter counts, so the survey's own comparison is not like-for-like
  [arxiv:2406.05615 tab. 3].
- **Headline TextVQA claim.** The 42.12% result uses ST-VQA annotation data.
  Without it the best PixelM4C result is 41.23% [arxiv:2105.05486 tab. 6].
  Compare like with like.

## Caveats

- No official code for PixelM4C was located, so the reported numbers cannot
  be checked against an implementation [arxiv:2105.05486 §5.2].
- The TextOCR annotations used as an OCR source in some rows are not a
  deployable setting; the authors say so [arxiv:2105.05486 §5.2].
- The AVSD numbers come from single runs with no variance reported, so gaps of
  a few tenths of MRR (for example 53.41 against 53.03) are not tested for
  significance [arxiv:1901.09107 tab. 3].
- The AVSD paper does not give its training hyperparameters, and the candidate
  recipe in its released code differs from the paper's description. Check the
  [AVSD card](/papers/1901.09107) before rebuilding its candidate sets
  [arxiv:1901.09107 §4 Selecting Candidate Answers].
- The survey's repository link is an awesome-style list, not an implementation
  [arxiv:2406.05615 §9].

## Papers

- [TextOCR/PixelM4C](/papers/2105.05486): TextOCR dataset and PixelM4C, a TextVQA model that uses a frozen TextOCR-trained spotter.
- [Audio-Visual Scene-Aware Dialog](/papers/1901.09107): AVSD benchmark and a late-fusion baseline that ranks 100 candidate answers, with dialog history, video and audio.
- [Video-language survey](/papers/2406.05615): survey of video-language tasks, architectures, training and data, with reported VideoQA scores.