ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

In plain words

Bar charts and line charts are everywhere, and people ask them questions that take several steps. A reader might have to find two values, subtract them, and then name the year with the biggest gap. Older chart-question datasets were written from templates, so their questions sounded unnatural, and their answers came from a short fixed list. That made them a poor test for this kind of reasoning.

This paper builds a benchmark of questions that people wrote about real charts from public sources. It also adds a larger set of questions generated from the charts' own written summaries. The baseline systems answer in two steps. First, they read a data table out of the chart picture. Then they combine that table with features of the image, so questions that refer to colours or shapes can be answered. When the true table is given, the systems do well. When the table must be read from the picture, accuracy drops, because reading charts is error-prone. Models score lower on this benchmark than on the older datasets.

TL;DR

• ChartQA is a benchmark with human-written questions (ChartQA-H) and machine-generated questions (ChartQA-M) about real charts §1 tab. 1.
• The best baseline, VisionTaPas, extends TaPas with image features, two extra operations (difference and ratio) and four cross-modality blocks §4.2.
• Author-reported ChartQA test accuracy for VisionTaPas: 45.52% with a table extracted from the chart, 61.84% with the gold table tab. 5.
• Use it if you need chart QA with multi-step arithmetic. Expect large gaps between gold and extracted tables, and check the metric (see Pitfalls).

Problem

Earlier chart-QA datasets (FigureQA, DVQA, LEAF-QA, LEAFQA++, PlotQA) use template-generated questions, charts drawn by software, and mostly fixed-vocabulary answers. Their questions often miss the multi-step arithmetic and the visual references that people actually use §1 tab. 1. PlotQA allows open answers but has no visual reasoning questions §2. Models that classify over a fixed answer set fail on open-vocabulary answers §1 §5.2.

The paper also argues that a table must be extracted from the chart image to answer questions when no table is given. Existing extraction methods do not attach values to their axis or legend labels §2 §4.1.

Dataset

Sources. Charts were crawled from Statista, Pew Research Center, Our World in Data (OWID) and OECD. For Pew, only chart images were kept. For the other three, the underlying data table, metadata, SVG and text description were extracted §3.1.

Annotation. Human-authored questions (ChartQA-H) came from Amazon Mechanical Turk. Each worker verified two earlier questions and wrote two new ones per chart, focusing on compositional questions (two or more operations such as sum, difference or average) and visual questions (references to colour, height or length of marks) §3.2 app. Additional Details on Data Annotation. Each question was answered by a second worker. Exact-match agreement was 61.04%. On 500 random samples, agreement rose to 78.55% when typos and lexical variants were allowed §3.2.

Generated questions (ChartQA-M). Two T5 models were fine-tuned on SQuAD. The first extracts candidate answers from a Statista human-written summary. The second writes a question for an answer given the summary. Questions whose answer does not appear in the chart's data table were dropped. Of 1,250 manually checked questions, 86.64% were complete, answerable and correct. The ChartQA-M test set was cleaned by hand §3.2.

Splits and counts. tab. 2 and tab. 3 give the split and source counts.

Question types in a random sample of 300 ChartQA-H pairs tab. 4:

Token statistics tab. 7 and visual-reference usage tab. 8 are not transcribed here.

Method

Two settings §4.1. Setting one gives the gold data table with each question. Setting two extracts the table from the chart image with an extended ChartOCR pipeline. The extracted table is the input to the answer models in both settings. Figure 2 shows the overall pipeline fig. 2.

Table extraction §4.1 app. Automatic Chart Data Extraction. The pipeline extends ChartOCR, which predicts mark values but not their labels. Its steps are:

# Chart-to-table extraction  [§4.1] [app. Automatic Chart Data Extraction]
# Input:  chart_image
# Output: data_table (rows: x labels, columns: series labels)

def extract_table(chart_image):
    # 1. Detect marks and text elements as keypoints  [app. Automatic...]
    marks, texts = keypoint_detector(chart_image)
    # 2. Read every detected text element with CRAFT  [§4.1]
    texts = [recognise(text) for text in texts]
    # 3. Convert mark geometry to values using the axis scale  [app. Auto...]
    values = scale_marks(marks, texts)
    # 4. Pair each value with its nearest x-axis label  [app. Automatic...]
    # 5. Pair each series with its legend label by colour, else by position
    return assemble_table(values, texts)

Answer models §4.2. Four models are compared, and all take the question and the table as text.

• T5 (encoder-decoder, text-to-text). The input is the question followed by the flattened table, and the model generates the answer §4.2.
• VL-T5. The T5 text input plus visual features of chart marks. Mask R-CNN with a ResNet-101 backbone detects 15 chart object classes. The features are padded with zeros to a fixed 36 objects §4.2 app. Visual Features Extraction in VL-T5.
• TaPas. A BERT-style encoder with row and column embeddings. The input is the question, then the flattened table. Two heads predict an aggregation operation (count, sum, average or none) and select cells §4.2.
• VisionTaPas (the paper's model). It has three parts: a ViT encoder for the image, a TaPas encoder for the question and table, and a cross-modality encoder with four blocks. Each block has a visual branch and a textual-tabular branch, each with multi-head cross-attention, then self-attention and a fully connected layer, with layer norm and residuals. The TaPas heads sit on the final textual-tabular layer §4.2 fig. 3.

The VisionTaPas operation head adds difference and ratio. It also adds classes for fixed-vocabulary answers such as yes and no §4.2.

# VisionTaPas answer step  [§4.2] [fig. 3]
# Input:  question tokens, table cells, chart image
# Output: answer (selected cells with an operation, or a class)

def answer_question(question, table, image):
    # 1. Encode the chart image into patch embeddings with ViT  [§4.2]
    visual = vit_encoder(image)                  # (n + 1, d)
    # 2. Encode the question and flattened table with TaPas  [§4.2]
    textual = tapas_encoder(question, table)     # (m + 1, d)
    # 3. Run the cross-modality blocks; each branch attends to the other  [§4.2]
    for block in cross_modality_blocks:
        visual, textual = block(visual, textual)
    # 4. Predict an operation from the [CLS] state  [§4.2]
    operation = operation_head(textual[0])
    # 5. Select cells and apply the operation to the selected values  [§4.2]
    cells = cell_selection_head(textual)
    return apply(operation, cells)

Weak supervision for difference and ratio §4.2. TaPas has no difference or ratio operation, and the paper supervises these by heuristic rather than from the final answer. The paper gives one example, for difference only. The heuristic searches the chart's values for a pair whose difference equals the answer.

# Weak cell supervision for "difference" questions  [§4.2]
# Input:  answer (number), data values of the chart
# Output: a pair of cells to supervise

def difference_supervision(answer, values):
    # 1. Try every pair of values from the table  [§4.2]
    for first, second in all_pairs(values):
        # 2. Keep the pair whose difference equals the answer  [§4.2]
        if abs(first - second) == answer:
            return (first, second)
    # Not stated: the paper does not say what happens when no pair matches

Training and inference. Fine-tuning settings are in the Hyperparameters section. Pretraining on PlotQA, then fine-tuning on ChartQA, was tested §5.2 tab. 6.

Hyperparameters

All values are from the paper's appendix unless marked otherwise [app. ChartQA Baseline Models]. The paper's display equations are unnumbered, so they are cited by section.

Not stated: learning-rate schedule and warmup, weight decay, optimiser, seeds, gradient clipping, and the number of attention heads.

Results

Author-reported. The benchmark is ChartQA. Accuracy is the fraction of questions answered correctly, using the relaxed rule for numbers §5.1. The models are VisionTaPas, VL-T5, T5, TaPas and the baselines PReFIL and PlotQA* §5.1.

Full results on the test sets tab. 5. Gold-table rows use the true table. Extracted rows use the table from §4.1. Bold marks the paper's bold entries.

Notes from the paper: the FigureQA test sets were not evaluated with gold tables, since they have none; FigureQA Val2 is used for the extraction accuracy tab. 5 app. Automatic Chart Data Extraction. STL-CQA and LEAF-NET are not publicly available; PReFIL and PlotQA* are the baselines used §5.1 (footnote). The paper calls the DVQA test sets Familiar and Novel.

Headline comparisons §5.2.

• VisionTaPas and VL-T5 come close to perfect with gold tables on earlier datasets. Accuracy drops when the table is extracted.
• DVQA extracted, test-novel with OCR: VisionTaPas 94.54%, which the paper reports as 14.5 points over PReFIL.
• PReFIL's OCR accuracy falls by 16.49 points against ORACLE. VisionTaPas falls by 0.92 points.
• ChartQA: VisionTaPas is best in both settings. PReFIL scores 4.8%, because it classifies over a fixed vocabulary. Accuracies on ChartQA are generally lower than on the earlier datasets §5.2.

Transfer and ablation tab. 6. ChartQA-H and ChartQA-M are the human and machine parts of the test set.

• Removing the difference and ratio supervision lowers overall accuracy by 1.80 points and ChartQA-H by 4.76 points §5.3.
• PlotQA pretraining raises VL-T5 from 41.56 to 51.84, but VisionTaPas only from 45.52 to 47.08 §5.2. The paper suggests VisionTaPas cannot represent nested operations, so pretraining helps less §5.2.

Breakdowns tab. 10 tab. 11. On the ChartQA test set, bar charts score highest. VisionTaPas: bar 49.80, line 38.20, pie 24.41, overall 45.52. VL-T5: bar 45.82, line 35.40, pie 25.00, overall 41.56 tab. 10. On 200 random ChartQA-H questions, data-retrieval questions score highest and visual questions lowest. VisionTaPas: data retrieval 60.00, visual-compositional 29.78, compositional 34.88, visual 16.21, overall 34.00. VL-T5: 50.00, 19.14, 24.41, 21.62, 26.50 tab. 11.

Extraction accuracy tab. 9. FigureQA 95.05, DVQA 89.98, PlotQA 80.88, ChartQA 83.85 app. Automatic Chart Data Extraction.

Error analysis §5.4 fig. 4. The paper describes three problems: nested operations, the separate treatment of table and image, and computer-vision errors in table extraction.

Paper vs code

Repository: vis-nlp/ChartQA, commit 044eabfc306abfe9340c5741f0093aefc5973d06, linked from the paper's introduction §1. The repository is GPL-3.0 (copyleft) LICENSE. Nothing from it is quoted in this card. The pseudocode is written in our own words and cited to code.

• Relaxed accuracy. The paper uses a 5% tolerance for numbers §5.1. The T5 and VL-T5 READMEs say their metric is exact accuracy and that the relaxed rule must be applied separately README.md README.md. A relaxed metric was not found in the repository.
• Table-extraction score. The distance function and the assignment step match the paper evaluate_data_extraction.py evaluate_data_extraction.py. The overall-score formula was not checked line by line.
• Difference and ratio. Answer types Ratio and Diff are handled question_answering_dataset.py. The code that generates the weak supervision was not located.
• Cross-modality layers. The paper uses 4 §4.2. The code default is 4 config.py.
• VisionTaPas training. The paper gives 30 epochs, batch 64 and LR 1e-5 for ChartQA app. ChartQA Baseline Models. The parser defaults are 4 epochs, batch 8, LR 1e-5, warmup 0.1 and weight decay 0.01 [code: Models/VisionTapas/args_parser.py#L23-L29]. The repository has no ChartQA run command, so the values used cannot be confirmed.
• VL-T5 training. The paper gives batch 96 and 30 epochs for ChartQA app. ChartQA Baseline Models. The launch script uses batch 32 and validation batch 64 VQA_VLT5.sh. It also passes --epochs 10 together with --load Epoch30 VQA_VLT5.sh VQA_VLT5.sh. The script's meaning is not documented. The LR (1e-4) and beam size (5) match the paper VQA_VLT5.sh VQA_VLT5.sh. The script also sets AdamW, warmup 0.05, gradient clipping 5 and fp16, none of which the paper states VQA_VLT5.sh VQA_VLT5.sh.
• VL-T5 defaults. Seed 9595, weight decay 0.01, dropout 0.1, and default epochs 12 are set in the parser param.py param.py param.py. The paper gives no seed.
• T5 training. The READMEs use learning rate 1e-4, 4 beams and 30 epochs, matching the paper README.md. The training batch is 8, against 40 in the paper README.md app. ChartQA Baseline Models.
• Mask R-CNN. The LR default is 0.00025 train.py. The COCO initialisation is set train.py. The learning rate is constant, with no decay steps train.py. The paper does not state a decay. Batch size and iteration count have no defaults in the code and must be passed on the command line train.py.
• Visual features. The paper pads to 36 §4.2. The code also truncates to 36 vqa_data.py, and zero-fills to 36 vqa_data.py.

Pitfalls

• Extracted tables are the realistic setting. Gold-table accuracy (61.84%) is much higher than extracted-table accuracy (45.52%) for VisionTaPas tab. 5. Report both, and say which one you used.
• Check the metric. Use the 5% relaxed rule for numbers, exact match otherwise §5.1. The repository's T5 and VL-T5 scripts use exact accuracy (see Paper vs code).
• Difference and ratio are weak supervision. On 100 sampled questions the heuristic was 24% noisy §4.2. Its ratio rule is not described. The paper supervises difference with the example "difference of 5 from values 3, 6, 8".
• Question-generation noise. Only 86.64% of 1,250 checked generated questions were correct, and only the ChartQA-M test set was cleaned by hand §3.2. Treat ChartQA-M training labels as noisy.
• Annotation agreement is modest. Exact-match agreement between workers was 61.04%; allowing typos and variants raised it to 78.55% on a 500-question sample §3.2.
• Unstated training details. The paper does not say whether the final models train on ChartQA-H and ChartQA-M together, nor the optimiser, warmup, weight decay, seeds, gradient clipping or head count.
• Numbers that do not add up. The introduction says 20,882 charts §1. The source counts in tab. 3 sum to 21,945 (4,804 human-source plus 17,141 machine-generated). Among bar charts, 79.4% simple plus 29.6% complex is over 100% §3.3. The visual-reference percentages in tab. 8 total 108.33%, and the paper does not state their base. Check these before reporting counts.
• Different test subsets. The 34.00% overall in tab. 11 comes from 200 sampled ChartQA-H questions. It is not directly comparable with the full-test figures in tab. 6.
• Equations are unnumbered. The distance and score formulas in the appendix have no equation numbers, so cite them by section app. Automatic Chart Data Extraction.
• Dataset licence. The repository's only licence file is GPL-3.0, which covers code. The charts come from Statista, Pew, OWID and OECD, each with its own terms. The paper says it used only charts that comply with those terms § Ethical Considerations. Check the terms before redistributing the chart images.

Lineage

• Builds on: TaPas (table question answering; WikiTQ pretraining) §4.2; T5 and VL-T5 §4.2; ViT §4.2; ChartOCR, extended for labelled tables §4.1; CRAFT for text recognition §4.1.
• Compared against: PReFIL (classification baseline), PlotQA* (PlotQA with extracted tables), FigureQA, DVQA, LEAF-QA and LEAFQA++ §5.1 tab. 1.
• Superseded by: not stated in the paper.