---
kind: concept
slug: chart-data-extraction
title: Chart data extraction
summary: >-
  Recovering a data table of values and labels from a chart image, so that code or a model can read the numbers. The core idea: detect marks and text, read the text, map mark positions to values through the axis scale, and pair values with their labels. The main trade-off: extraction errors lower downstream accuracy, for example from 61.84% with gold tables to 45.52% with extracted tables.
tags: [chart-understanding, table-extraction, ocr, vision-language]
papers: ["2203.10244"]
related: [dataset-curation, vision-transformer, multimodal-language-model, vision-language-pretraining]
status: draft
updated_at: 2026-10-19
---

## Overview

A bar, line or pie chart holds numbers, but only as marks on a picture. To
answer a question about the numbers, or to feed them to other code, the marks
must be turned back into a table of values with their axis and legend labels.
The output has one row per x-axis label and one column per series label
[arxiv:2203.10244 §4.1] [arxiv:2203.10244 app. Automatic Chart Data Extraction].

The invariant that every variant shares is that the values must be read from
the picture and attached to the right labels. Within the ChartQA paper, the
extracted table is the input to the answer models, so extraction errors reach
the final answers [arxiv:2203.10244 §4.2].

## Variants

### Extended ChartOCR pipeline (labelled tables)

The paper extends ChartOCR, which predicts mark values but not their labels.
The extension detects marks and text as keypoints, reads the text, converts mark
geometry into values with the axis scale, pairs each value with its nearest
x-axis label, and pairs each series with a legend label by colour, or by
position when colour does not decide [arxiv:2203.10244 §4.1]
[arxiv:2203.10244 app. Automatic Chart Data Extraction]. Card:
[ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](/papers/2203.10244).

Reported extraction accuracy with this pipeline: FigureQA 95.05, DVQA 89.98,
PlotQA 80.88 and ChartQA 83.85 [arxiv:2203.10244 tab. 9]. The paper does not
say how it handles a label that is missing from the chart, or a chart where
colour and position both fail to match a legend entry.

### No extraction: use the source table

When the chart's underlying data table is available, the paper's first setting
gives the true table with each question and skips extraction
[arxiv:2203.10244 §4.1]. This is an upper bound for any QA step built on the
table, not an extraction method. It is useful as a baseline for measuring how
much extraction costs.

## When to use which

| Situation | Use | Why |
|---|---|---|
| The source data table exists | Use the table directly; skip extraction | With gold tables, VisionTaPas reaches 61.84% on ChartQA test, against 45.52% with extracted tables [arxiv:2203.10244 tab. 5] |
| Only the chart image, and you need values with their labels | Extended ChartOCR pipeline with label pairing | The labels are attached by the extension, which ChartOCR alone does not do [arxiv:2203.10244 §4.1]. Extraction accuracy on ChartQA is 83.85 [arxiv:2203.10244 tab. 9] |
| Only the chart image, and the goal is answering questions | Extract a table, then answer with a model that also sees the image | On ChartQA test, VisionTaPas (with image features) scores 45.52% and VL-T5 scores 41.56%, both with extracted tables [arxiv:2203.10244 tab. 5] |
| The questions need open-vocabulary answers | Avoid a fixed answer-class head | PReFIL classifies over a fixed vocabulary and scores 4.8% on ChartQA test [arxiv:2203.10244 tab. 5] [arxiv:2203.10244 §5.2] |
| Limited build effort | Expect several trained parts | The pipeline combines a keypoint detector, a text reader and axis-scale logic [arxiv:2203.10244 §4.1]. The VL-T5 variant also needs a Mask R-CNN trained on 15 chart object classes [arxiv:2203.10244 §4.2] |

## Implementation sketch

The steps below follow the extraction pipeline in the paper. Details such as the
detector and the text reader are paper-specific; see the card for the sources.

```pseudocode
# Chart image to labelled data table  [arxiv:2203.10244 §4.1]
# Input:  chart_image
# Output: table with rows = x labels, columns = series labels

def extract_table(chart_image):
    # 1. Detect mark keypoints and text regions in the image  [§4.1]
    marks, texts = keypoint_detector(chart_image)
    # 2. Read the text of every detected region  [§4.1]
    texts = [read_text(region) for region in texts]
    # 3. Convert mark positions to values with the axis scale  [app. Auto...]
    values = scale_marks(marks, texts)
    # 4. Attach each value to its nearest x-axis label  [app. Auto...]
    # 5. Attach each series to a legend label by colour, else by position  [§4.1]
    return assemble_table(values, texts)
```

The table extraction score matches each predicted value to a gold value. Its
distance is min(1, |(gt − pr) / gt|), and the assignment cost is taken over
K = max(N, M) entries [arxiv:2203.10244 app. Automatic Chart Data Extraction].
Use it for evaluation, not as the answer.

## Contradictions

None known. Only one card in the collection covers this concept. Within that
paper, the counts disagree, as noted in Pitfalls.

## Pitfalls

- **Gold and extracted numbers differ.** Accuracy with gold tables is much
  higher than with extracted tables, for every model in the paper's table
  [arxiv:2203.10244 tab. 5]. Report which table you used.
- **Check the metric.** The paper uses a 5% tolerance for numeric answers and
  exact match otherwise [arxiv:2203.10244 §5.1]. The paper's repository, which
  is GPL-3.0, uses exact accuracy in its T5 and VL-T5 scripts, and no relaxed
  metric was found there. Implement from the description, not by copying.
- **Pie and line charts score lower.** For VisionTaPas, bar charts score 49.80,
  line charts 38.20 and pie charts 24.41 on the ChartQA test set
  [arxiv:2203.10244 tab. 10]. These are QA scores, not extraction scores.
- **Extraction is a weak point for questions that need visual reasoning.** On
  200 sampled ChartQA-H questions, visual questions scored 16.21 for VisionTaPas
  [arxiv:2203.10244 tab. 11].
- **Counts in the paper do not add up.** The introduction gives 20,882 charts
  [arxiv:2203.10244 §1]. The source counts in the dataset table sum to 21,945
  [arxiv:2203.10244 tab. 3]. Check counts before you report them.
- **Chart sources have their own terms.** The charts come from Statista, Pew,
  OWID and OECD. The paper says it used only charts that comply with those
  terms [arxiv:2203.10244 §Ethical Considerations]. Check them before you
  redistribute chart images.

## Papers

- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](/papers/2203.10244): a benchmark with a chart-to-table extraction pipeline that extends ChartOCR with label pairing, and two baselines that answer from the extracted table and the chart image.