Medical Visual Question Answering: A Survey

In plain words

Medical visual question answering is the task of giving a doctor-style answer to a written question about a medical picture, such as an X-ray, a CT scan or a tissue slide. This survey gathers the public datasets built for the task and the research systems that have been built on them, then sorts those systems by their parts.

Most systems look the same at a high level: one part reads the picture, one part reads the question, a third part combines the two, and a last part picks or writes the answer. The authors find that most of the effort has gone into the picture-reading part, because medical image datasets are small. They also list open problems: questions that real clinicians and patients would ask, using extra patient information, showing why an answer is right, working on different hospitals' data, and fitting into clinical workflows.

TL;DR

The survey covers eight public medical VQA datasets from 2018 to 2021 and 45 method papers that describe 46 approaches §2.1 §3. Most approaches use an image encoder, a question encoder, a fusion step and an answer head §3.1. Image encoders are the main area of work, and attention-based fusion scored higher on average than non-attention fusion in the reviewed results §3.7. Six challenges are proposed for future work §5.

Problem

Medical VQA needs expert-written question-answer pairs, which are costly to collect, so most public datasets are small or built by templates §1 §2.1. The authors argue that general-domain VQA methods do not transfer directly, because medical images are fine-grained, medical language is specialised and the answers have to be reliable §1 §4.

The survey also questions whether BLEU, a common text-overlap score, fits short medical answers such as yes/no or single phrases §2.2.

Method

This is a literature survey, so it has no training method and no equations. The review method is as follows §3:

1. Dataset survey: the authors searched for medical VQA datasets through Google Scholar, medical imaging venues and top journals. Ten dataset papers were found; two were earlier versions of others, leaving eight datasets §1 §2.1.
2. Method search, in two strands. (a) The ImageCLEF VQA-Med overview papers were used to find participating teams, giving 32 work notes. (b) A citation search of papers that cite the datasets returned 217 papers, of which 12 were kept §3 fig. 2. The survey reports 45 papers in total, counting 32 work notes and 13 conference or journal papers §3.
3. Each approach was classified by its image encoder, its pre-training source, its language encoder, its fusion method, its output mode and any extra techniques. These attributes are the columns of tab. 3.

The general architecture the survey describes for most of the 46 approaches is the joint-embedding framework, which the authors trace to the original VQA baseline §3.1 fig. 3. Its four parts are:

• an image encoder, usually a CNN such as VGG or ResNet, often pre-trained §3.1 §3.2;
• a question encoder, such as an LSTM, a GRU or a Transformer with BERT or BioBERT weights §3.3;
• a fusion step, often element-wise product, concatenation, pooling or attention §3.4;
• an answer head, either a classifier over a fixed answer set or a text generator §3.5.

Thirty-nine of the 46 approaches fit this framework. The other seven use only the image to answer §3.1.

# Joint-embedding medical VQA, as described in the survey  [§3.1] [fig. 3]
# Input:  image (H, W, 3); question (string)
# Output: answer (class index or generated text)

def answer(image, question):
    # 1. Encode the image, often with ImageNet-pretrained weights  [§3.2]
    visual = image_encoder(image)            # (C, h, w) or (d,)
    # 2. Encode the question  [§3.3]
    language = question_encoder(question)    # (d,)
    # 3. Fuse the two features  [§3.4]
    fused = fusion(visual, language)         # e.g. attention plus product
    # 4. Classify over a fixed answer set, or generate text  [§3.5]
    return answering_head(fused)

Other techniques the survey counts across the reviewed approaches are sub-task routing (a question-type or modality classifier that picks a specialised model), global average pooling in place of fully connected layers, and auxiliary pre-training on captions, segmentation or unlabelled images §3.2 §3.6.

Hyperparameters

The survey does not propose a training setup, so it states no hyperparameters §3. Per-approach learning rates, epochs and batch sizes are not stated in the survey tab. 3. Check the original paper of any approach before reusing its setup.

Results

Author-reported in the original papers, compiled by the survey. Each row is the top entry the survey lists for that dataset, with the source's own setup. The survey does not re-run any method tab. 3. Starred rows (*) were not proposed during the challenge period tab. 3.

The survey's summary points are these. Among VQA-Med-2019 approaches, BERT-based language encoders outperformed LSTM-based ones §3.7. The survey attributes the top VQA-Med-2020 and 2021 results to ensembles of image encoders without a language encoder §3.7. The top three VQA-RAD teams all used extra image-encoder pre-training §3.7. Among VQA-Med-2019 entries, approaches with attention fusion averaged 0.576 accuracy against 0.522 for those without §3.7.

Dataset statistics as the survey lists them tab. 1:

Image counts range from 315 to 91,060 across medical datasets, and most are under 5,000 images §2.1 tab. 1. The survey notes that VQA-RAD is the only dataset built from natural clinician questions §2.1.

Paper vs code

No official code. The paper links no code repository; its only URLs are ImageCLEF, PubMed Central and footnoted tool pages. A GitHub repository search for the survey's title found linzhlalala/Medical-Visual-Question-Answering-Survey, which has no description and no license, and cloneiq/ARISE-MedVQA, a curated literature hub. Neither is attributable to the authors and neither contains method code, so neither was cloned.

The survey itself describes no software. The per-approach code of the 46 reviewed methods is outside this card.

Pitfalls

• Counts do not match inside the paper. The introduction says three of the datasets are ImageCLEF competitions §1, but Table 1 lists four VQA-Med editions (2018, 2019, 2020, 2021) tab. 1.
• The citation search is described as returning 12 kept papers §3. Add 32 work notes and 12 papers and you get 44, not 45. The survey then reports 13 conference or journal papers §1 §3.
• The survey says 45 papers describe 46 approaches §3, and gives 39 + 7 = 46 for the framework split §3.1. An agent counting "papers" and "approaches" must choose one.
• Table 3 mixes metrics and subsets. Some BLEU values are on subsets (MedFuseNet for VQA-Med-2019 and PathVQA), and the MEVF VQA-RAD cell lists two numbers and a bracketed third tab. 3. Do not sort the table as one ranking.
• Starred rows (*) were not part of the challenge period tab. 3. Their scores are not directly comparable with challenge-period scores.
• The claim that BERT-based encoders "significantly" beat LSTM encoders on VQA-Med-2019 is a descriptive reading of the table with no statistical test §3.7.
• BLEU is criticised as a metric for short medical answers §2.2, but the survey gives no alternative metric with reported numbers.
• The survey cites the VQA-Med-2021 overview with two different references (abacha2020overview in §2.1 and ImageCLEF-VQA-Med2021 in Table 1) §2.1 tab. 1.
• The survey's five-part challenge list is its own argument, not measured results §5. For instance, the claim that no existing study quantifies medical VQA interpretability is a statement about the authors' search §5.3.4.
• The reviewed papers mostly use ImageNet pre-training, which the survey calls a poor fit for medical images, though workable when labels are scarce §3.2.
• Figures in the survey show dataset samples and encoder composition fig. 1 fig. 4; they are not reproduced here.

Lineage

• Builds on: the joint-embedding VQA baseline (LSTM Q+I) of Antol et al., cited by the survey as antol2015vqa §3.1. The survey does not state its arXiv identifier, so none is given here.
• Compared against: general-domain VQA datasets (Visual Genome, VQA 2.0, OK-VQA) and their methods §2.1 §4 tab. 1.
• Superseded by: not stated.