Visual question answering

Overview

Visual question answering (VQA) gives a model an image or a video and a question in text, and asks for an answer in text. The answer is either picked from a fixed vocabulary, picked from a list of candidates, or generated as free text. Three families appear in the current cards:

• Scene-text VQA, where the question concerns words written in the scene (signs, labels, posters). The answer model needs an OCR component to read those words 2105.05486 §1.
• Video question answering, where the model must reason over frames in order, not over a single picture 2406.05615 §2.
• Dialog-based video QA, where the question is one turn of a conversation about a video, so earlier turns are part of the input 1901.09107 §1.

The evidence base is still thin: three cards list this slug. For generic VQA evaluation, see Visual question-answering evaluation.

Variants

Scene-text VQA: PixelM4C

The answer model is M4C, an iterative answer-prediction transformer for TextVQA 2105.05486 §5.2. PixelM4C keeps M4C and changes its text source. It runs a TextOCR-trained Mask TextSpotter v3 as a frozen module and passes the spotter's recognised tokens, with features taken from the decoder's last hidden state at the end-of-sequence token, into M4C 2105.05486 §5.2 tab. 6. Card: TextOCR/PixelM4C card.

Reported TextVQA val accuracy, author-reported 2105.05486 tab. 6:

• without ST-VQA data: PixelM4C 41.23 against M4C 39.40;
• with ST-VQA data: PixelM4C 42.12 against M4C 40.55.

The same paper's TextCaps variant has a lower CIDEr than its baseline (85.32 against 89.60), and the authors report no significant gain there except on BLEU-4 2105.05486 §5.2 tab. 6.

Dialog-based video QA: AVSD baseline

The AVSD paper defines a benchmark of 11,816 dialogs about Charades videos, each with ten question-answer rounds and a closing summary 1901.09107 §3.1. Its baseline ranks answers instead of generating them. The authors chose this because discriminative ranking worked better than generation in visual dialog 1901.09107 §4 Decoder Model. The model works as follows:

• an LSTM encodes the script and the earlier rounds, and a second LSTM encodes the current question;
• the video and the audio each enter as one fixed-size vector, from I3D and AENet respectively 1901.09107 §4 Implementation Details;
• the four vectors are concatenated and passed through one tanh layer;
• each of 100 candidate answers is encoded and scored by an inner product with the fused vector, and a softmax over the candidates gives the training loss 1901.09107 eq. 1.

Card: Audio-Visual Scene-Aware Dialog.

Reported test MRR, author-reported, single runs 1901.09107 tab. 3:

• question only: 36.12;
• question plus dialog history: 50.40;
• question, dialog history and video: 53.41;
• the same with audio added: 53.03.

Shuffling the order of the earlier rounds lowers MRR for each row, but the model was tested on shuffled dialogs without being retrained on them 1901.09107 tab. 4. For the audio-related questions, mean rank is 6.91 with the question alone and 6.69 when audio is added 1901.09107 tab. 5.

Video question answering

The survey splits the methods by architecture 2406.05615 §4:

• Pre-Transformer models use CNN or RNN encoders and fusion by attention, multiplication or graphs. The best listed MSRVTT accuracy in this family is 40.0 (QueST) 2406.05615 tab. 3.
• Transformer-based models come in shared, stacked and dual forms. The best listed MSRVTT accuracy is 45.0 (LAVENDER, stacked) [arxiv:2406.05615 tab. 3].
• LLM-augmented models pass visual tokens, produced by a semantic translator (a Q-Former with learnable queries or a linear projection), into an LLM that generates the answer 2406.05615 §4.3. The best listed MSRVTT accuracy is 58.9 (LLaMA-Vid, EVA-G with Vicuna) [arxiv:2406.05615 tab. 3].

All these scores are copied from the cited papers. The survey does not say whether they were re-run, and the backbones differ 2406.05615 tab. 3. Card: video-language survey card.

When to use which

The cards give no measured latency, memory or data-efficiency numbers for VQA. Pick a design from the constraints above and check its cost in your own setting.

Implementation sketch

The common core: encode the image (or frames), encode the question, combine the two, and produce an answer. Where variants differ, the notes say how.

# Generic VQA step  [arxiv:2406.05615 §4.3] [arxiv:2105.05486 §5.2]
#                   [arxiv:1901.09107 §4]
# Input:  image or frames; question text; optional dialog history
# Output: answer text

def answer(visual_input, question, history=None):
    # 1. Encode visual input (CNN, ViT, frozen detector, or fixed-size I3D vector)
    visual_tokens = visual_encoder(visual_input)
    # 2. Optional: read text in the scene; PixelM4C keeps tokens with
    #    recognition score >= 0.2 and caps them at 200  [arxiv:2105.05486 §5.2]
    ocr_tokens = scene_text_reader(visual_input)   # frozen in PixelM4C
    # 3. Optional: encode earlier dialog rounds  [arxiv:1901.09107 §4]
    history_state = encode_history(history) if history else None
    # 4. Fuse question, visual tokens and OCR tokens
    #    Fusion can be an iterative transformer (M4C) or an LLM prompt
    #    [arxiv:2406.05615 §4.3]; AVSD concatenates and applies one tanh layer
    fused = fuse(question, visual_tokens, ocr_tokens, history_state)
    # 5. Predict: score a fixed vocabulary, rank a candidate list, or
    #    generate free text  [arxiv:1901.09107 §4 Decoder Model]
    return decode_answer(fused)

Contradictions

• Video QA ranking. The survey says LLM-augmented models have begun to outperform Transformer-based models on VideoQA 2406.05615 §4.4. The table rows support that direction but use different backbones and parameter counts, so the survey's own comparison is not like-for-like 2406.05615 tab. 3.
• Headline TextVQA claim. The 42.12% result uses ST-VQA annotation data. Without it the best PixelM4C result is 41.23% 2105.05486 tab. 6. Compare like with like.

Caveats

• No official code for PixelM4C was located, so the reported numbers cannot be checked against an implementation 2105.05486 §5.2.
• The TextOCR annotations used as an OCR source in some rows are not a deployable setting; the authors say so 2105.05486 §5.2.
• The AVSD numbers come from single runs with no variance reported, so gaps of a few tenths of MRR (for example 53.41 against 53.03) are not tested for significance 1901.09107 tab. 3.
• The AVSD paper does not give its training hyperparameters, and the candidate recipe in its released code differs from the paper's description. Check the AVSD card before rebuilding its candidate sets 1901.09107 §4 Selecting Candidate Answers.
• The survey's repository link is an awesome-style list, not an implementation 2406.05615 §9.

Papers

• TextOCR/PixelM4C: TextOCR dataset and PixelM4C, a TextVQA model that uses a frozen TextOCR-trained spotter.
• Audio-Visual Scene-Aware Dialog: AVSD benchmark and a late-fusion baseline that ranks 100 candidate answers, with dialog history, video and audio.
• Video-language survey: survey of video-language tasks, architectures, training and data, with reported VideoQA scores.