Table question answering

Overview

Given a question and a table, return the answer. The answer is a cell value, a number computed from cells, or a short text. Every variant reads the table cells as input, either as flattened text, as encoded rows, or through a program or table helpers that run over the table 2203.10244 §4.2 2005.08314 §3.1 2210.02875 §2.1 2304.09842 §4.3. When only a chart image is given, a table must first be recovered from it 2203.10244 §4.1.

Variants

Fine-tuned cell selection and aggregation (TaPas-style)

A BERT-style encoder reads the question, then the flattened table with row and column embeddings. Two heads predict an aggregation operation (count, sum, average or none) and select cells. The model is trained on the task data 2203.10244 §4.2. On ChartQA with extracted tables it reaches 41.28 test accuracy, and 51.80 with gold tables 2203.10244 tab. 5. Card: ChartQA, which uses it as a baseline.

Fine-tuned text generation (T5, VL-T5)

An encoder-decoder reads the question and the flattened table as text and generates the answer 2203.10244 §4.2. VL-T5 adds visual features of the chart marks to the same input. On ChartQA with extracted tables the two reach 41.04 (T5) and 41.56 (VL-T5) test accuracy. With gold tables they reach 59.80 and 59.12 2203.10244 tab. 5. PlotQA pretraining raises VL-T5 to 51.84 on the same test set 2203.10244 tab. 6. Card: ChartQA.

Chart image with an extracted table (VisionTaPas)

For questions about a chart. A ViT encodes the image, a TaPas encoder encodes the question and table, and four cross-modality blocks let each branch attend to the other. The operation head adds difference and ratio, and the cell heads sit on the textual branch 2203.10244 §4.2. It is the best of the ChartQA baselines, at 45.52 test accuracy with extracted tables and 61.84 with gold tables 2203.10244 tab. 5. Removing the difference and ratio supervision lowers overall accuracy by 1.80 points 2203.10244 §5.3. Card: ChartQA.

Training-free program writing (Binder)

The LM writes a SQL or Python program for the question. Where the language cannot express a step, the program contains a call back to the same LM over a few cells or columns. Each program runs, and the most frequent answer wins. The method uses 14 annotated exemplars and no training 2210.02875 §2.1 2210.02875 §3.1. Author-reported results, WikiTQ test and TabFact small test: Binder reaches 64.6 and 85.1 with fixed exemplars, and 86.0 on TabFact with a retriever. SQL-only programs reach 61.1 and 80.7. Direct LM prompting with the table reaches 48.7 and 72.6 2210.02875 §3.2. Card: Binder.

Planner-chosen table tools (Chameleon)

A planner LLM writes a short plan that names modules, and the modules run in order, each passing its output on through an input update and a shared cache. Nothing is trained; the planner works from module descriptions and a few demonstrations 2304.09842 §3 2304.09842 eq. 1. For math word problems over tables, the module set includes row and column lookups that cut the table down, a table verbaliser, and Python program generation, verification and execution 2304.09842 §4.3 2304.09842 tab. 9. The planner may choose from these modules, and a plan that breaks the TabMWP ordering rules falls back to a fixed program-writing sequence 2304.09842 §4.3. Author-reported TabMWP test accuracy is 98.78 with GPT-4 and 93.28 with ChatGPT 2304.09842 tab. 4. On a 500-example ChatGPT subset, disabling Program Generator costs 7.4 points and disabling Table Verbalizer 0.2 2304.09842 tab. 5. Card: Chameleon.

Table encoder inside a semantic parser (TaBERT)

Not a question answerer on its own. TaBERT is a BERT-initialized encoder, pretrained on 26.6M tables with nearby text. It reads the question with K rows of the table, and vertical self-attention lets those rows share information 2005.08314 §3.1 2005.08314 app. A.1. A parser then consumes its column and token vectors. In the paper the parser is MAPO for WikiTableQuestions and TranX for Spider 2005.08314 §4.1 2005.08314 §4.2. On WTQ with K = 3 and the Large encoder, the run that was best on dev reaches 52.3 test accuracy. The mean over runs is 51.8 ± 0.6, against 46.9 for the previous ensemble 2005.08314 tab. 1 2005.08314 §5. On Spider dev the same setup reaches 64.5 ± 0.6 mean, below RyanSQL + BERT at 66.6 2005.08314 tab. 2. Card: TaBERT.

When to use which

Two cautions for Binder. Its reported numbers use code-davinci-002 2210.02875 §3.1, but the repository's default engine is a chat model without logprobs, so the numbers may not transfer. Generated programs are executed with Python eval, so run them in a sandbox. See the Binder card, Paper vs code and Pitfalls.

Chameleon parses its planner's output with Python eval too. Use a strict parser with a whitelist of module names before reusing that design. See the Chameleon card, Pitfalls.

The TaBERT code is CC BY-NC 4.0, so it cannot be reused commercially. The README links pretrained weights but states no separate licence for them. Re-implement from the paper, and see the TaBERT card, Paper vs code and Pitfalls.

Implementation sketch

The common core is one function with a branch per variant. The variants differ only in the branches and in where the table comes from. The TaBERT route does not return an answer: it returns column and token vectors for a parser, and its pseudocode is on the TaBERT card 2005.08314 §3.1.

# Shared shape of table QA  [arxiv:2210.02875 §2.1] [arxiv:2203.10244 §4.1]
# Input:  question, table (or a chart image to extract it from)
# Output: answer

def table_qa(question, table, image=None, variant="program"):
    # Only a chart image is given: recover a table first  [arxiv:2203.10244 §4.1]
    if table is None:
        table = extract_table(image)
    # Cell selection and operation  [arxiv:2203.10244 §4.2]
    # VisionTaPas also encodes the image; omitted here
    if variant == "select":
        return apply_operation(predict_operation(question, table),
                               select_cells(question, table))
    # Answer text generated from the flattened table  [arxiv:2203.10244 §4.2]
    if variant == "generate":
        return generate(flatten(question, table))
    # Planner names a module chain, then the modules run in order  [arxiv:2304.09842 §3]
    if variant == "plan":
        modules = planner_llm(question, table, module_descriptions)
        return run_chain(modules, question, table)
    # Sample programs, run each one, and vote  [arxiv:2210.02875 §2.2]
    # n = 20 for WikiTQ  [arxiv:2210.02875 app. A.2]
    programs = lm_sample(question, table, n=20)
    return majority_vote([run_program(p, table) for p in programs])

Contradictions

• WikiTQ ranking. Under the paper's semantic-match evaluator, Binder (64.6 test) is above OmniTab (63.3 test) 2210.02875 §3.2. Under the official evaluator, Binder reaches 61.9 test and OmniTab 62.9 test 2210.02875 app. C.2. The ranking flips with the evaluator, so state which one a number uses.
• Share of program-unsolvable questions. The text says "about 20%" 2210.02875 §3.1. The authors' re-derived split gives 44.19% and the SQUALL annotation gives 22.08% 2210.02875 app. C.4. Subset results depend on which split is used.

The WTQ numbers of TaBERT and Binder come from different setups: a trained MAPO parser against a training-free LM with exemplars. They are not one scale and should not be ranked against each other. Likewise, the ChartQA figures are not comparable with WTQ or TabFact. The TabMWP figures of Chameleon belong to a third benchmark, and its scoring accepts a prediction that rounds to the reference value Chameleon card, Pitfalls. Do not rank them against the WTQ or TabFact numbers above.

Papers

• Binding Language Models in Symbolic Languages: training-free SQL or Python programs with LM calls inside, voted over samples.
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models: frozen LLM planner that chains table, search, captioning and program modules, with few-shot prompts and no training.
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning: chart QA benchmark with TaPas, T5, VL-T5 and VisionTaPas baselines on extracted and gold tables.
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data: BERT-style table encoder pretrained on tables with text, plugged into WTQ and Spider parsers.