wikillm

Concept · 1 paper · updated 2026-10-19

Chart data extraction

Recovering a data table of values and labels from a chart image, so that code or a model can read the numbers. The core idea: detect marks and text, read the text, map mark positions to values through the axis scale, and pair values with their labels. The main trade-off: extraction errors lower downstream accuracy, for example from 61.84% with gold tables to 45.52% with extracted tables.

chart-understandingtable-extractionocrvision-language

Markdown4 min read

AI-compiled overview, updated 2026-10-19, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A bar, line or pie chart holds numbers, but only as marks on a picture. To answer a question about the numbers, or to feed them to other code, the marks must be turned back into a table of values with their axis and legend labels. The output has one row per x-axis label and one column per series label 2203.10244 §4.1 2203.10244 app. Automatic Chart Data Extraction.

The invariant that every variant shares is that the values must be read from the picture and attached to the right labels. Within the ChartQA paper, the extracted table is the input to the answer models, so extraction errors reach the final answers 2203.10244 §4.2.

Variants

Extended ChartOCR pipeline (labelled tables)

The paper extends ChartOCR, which predicts mark values but not their labels. The extension detects marks and text as keypoints, reads the text, converts mark geometry into values with the axis scale, pairs each value with its nearest x-axis label, and pairs each series with a legend label by colour, or by position when colour does not decide 2203.10244 §4.1 2203.10244 app. Automatic Chart Data Extraction. Card: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.

Reported extraction accuracy with this pipeline: FigureQA 95.05, DVQA 89.98, PlotQA 80.88 and ChartQA 83.85 2203.10244 tab. 9. The paper does not say how it handles a label that is missing from the chart, or a chart where colour and position both fail to match a legend entry.

No extraction: use the source table

When the chart's underlying data table is available, the paper's first setting gives the true table with each question and skips extraction 2203.10244 §4.1. This is an upper bound for any QA step built on the table, not an extraction method. It is useful as a baseline for measuring how much extraction costs.

When to use which

SituationUseWhy
The source data table existsUse the table directly; skip extractionWith gold tables, VisionTaPas reaches 61.84% on ChartQA test, against 45.52% with extracted tables 2203.10244 tab. 5
Only the chart image, and you need values with their labelsExtended ChartOCR pipeline with label pairingThe labels are attached by the extension, which ChartOCR alone does not do 2203.10244 §4.1. Extraction accuracy on ChartQA is 83.85 2203.10244 tab. 9
Only the chart image, and the goal is answering questionsExtract a table, then answer with a model that also sees the imageOn ChartQA test, VisionTaPas (with image features) scores 45.52% and VL-T5 scores 41.56%, both with extracted tables 2203.10244 tab. 5
The questions need open-vocabulary answersAvoid a fixed answer-class headPReFIL classifies over a fixed vocabulary and scores 4.8% on ChartQA test 2203.10244 tab. 5 2203.10244 §5.2
Limited build effortExpect several trained partsThe pipeline combines a keypoint detector, a text reader and axis-scale logic 2203.10244 §4.1. The VL-T5 variant also needs a Mask R-CNN trained on 15 chart object classes 2203.10244 §4.2

Implementation sketch

The steps below follow the extraction pipeline in the paper. Details such as the detector and the text reader are paper-specific; see the card for the sources.

Pseudocode
# Chart image to labelled data table  [arxiv:2203.10244 §4.1]
# Input:  chart_image
# Output: table with rows = x labels, columns = series labels

def extract_table(chart_image):
    # 1. Detect mark keypoints and text regions in the image  [§4.1]
    marks, texts = keypoint_detector(chart_image)
    # 2. Read the text of every detected region  [§4.1]
    texts = [read_text(region) for region in texts]
    # 3. Convert mark positions to values with the axis scale  [app. Auto...]
    values = scale_marks(marks, texts)
    # 4. Attach each value to its nearest x-axis label  [app. Auto...]
    # 5. Attach each series to a legend label by colour, else by position  [§4.1]
    return assemble_table(values, texts)

The table extraction score matches each predicted value to a gold value. Its distance is min(1, |(gt − pr) / gt|), and the assignment cost is taken over K = max(N, M) entries 2203.10244 app. Automatic Chart Data Extraction. Use it for evaluation, not as the answer.

Contradictions

None known. Only one card in the collection covers this concept. Within that paper, the counts disagree, as noted in Pitfalls.

Pitfalls

  • Gold and extracted numbers differ. Accuracy with gold tables is much higher than with extracted tables, for every model in the paper's table 2203.10244 tab. 5. Report which table you used.
  • Check the metric. The paper uses a 5% tolerance for numeric answers and exact match otherwise 2203.10244 §5.1. The paper's repository, which is GPL-3.0, uses exact accuracy in its T5 and VL-T5 scripts, and no relaxed metric was found there. Implement from the description, not by copying.
  • Pie and line charts score lower. For VisionTaPas, bar charts score 49.80, line charts 38.20 and pie charts 24.41 on the ChartQA test set 2203.10244 tab. 10. These are QA scores, not extraction scores.
  • Extraction is a weak point for questions that need visual reasoning. On 200 sampled ChartQA-H questions, visual questions scored 16.21 for VisionTaPas 2203.10244 tab. 11.
  • Counts in the paper do not add up. The introduction gives 20,882 charts 2203.10244 §1. The source counts in the dataset table sum to 21,945 2203.10244 tab. 3. Check counts before you report them.
  • Chart sources have their own terms. The charts come from Statista, Pew, OWID and OECD. The paper says it used only charts that comply with those terms 2203.10244 §Ethical Considerations. Check them before you redistribute chart images.

Papers

Related

ConceptDataset curationHow training and evaluation data is sourced, filtered, deduplicated, labelled and documented. Covers web text, image-text pairs, speech, vision, 3D objects, parallel text, robot demonstrations, instruction and benchmark data, the filters each paper used and their reported effects, and the licence and documentation traps that come with reusing them.ConceptVision Transformer (ViT) backbonesA Vision Transformer cuts an image into patches, embeds each as a token and runs a Transformer encoder over them. Variants change the pretraining objective, the token mixer, the input or the role in a larger system. Frozen features often transfer to dense and specialised tasks. Trade-offs: results depend on pre-training, and attention cost grows with the square of the token count.ConceptMultimodal language modelA language model that reads images, video, audio or other signals next to text and answers in text. Designs differ in how the non-text input enters the model, which parts train, and whether the output can also be masks or boxes: a frozen LLM behind a small bridge, a natively multimodal decoder, a fully fine-tuned generalist, or one token vocabulary for everything.ConceptVision-language pre-trainingVision-language pre-training ties images to text. Papers here train encoders with contrastive or align-then-fuse losses, bootstrap captions, train one model to read images and write text, bridge frozen encoders to LLMs, or use a frozen CLIP as a loss, prior or weight-space target. Main trade-off: data and compute cost against per-sample cost and licences.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.