FeTaQA: Free-form Table Question Answering

In plain words

Most systems that answer questions from a table return a short value: a number, a name or one cell. This paper argues that many real questions need a short explanation assembled from several parts of a table. To test this, the authors built a dataset of questions about Wikipedia tables, each with a sentence-length answer written by a person and the cells that support it. Its answers are explanations, not spans copied from the source.

Two baselines try the task. One chains a model that picks the relevant cells with a text writer that turns those cells into a sentence. The other flattens the whole table into a line of text, feeds it with the question to one text-to-text model, and lets that model write the answer. The second approach does much better here, and both are far below the human answers. Judges rate generated answers as correct much less often than the human references.

The point for an engineer is that a system which returns only a cell cannot explain its answer, and this dataset measures whether a model can write one.

TL;DR

FeTaQA has 10,330 Wikipedia table questions with free-form, human-written answers and the supporting highlighted cells, split 7,326 / 1,001 / 2,003 §2.3 tab. 2. Two baselines are reported: a TAPAS-based cell retriever with a T5 data-to-text writer (pipeline) and an end-to-end T5 that reads a linearised table. The best author-reported test score is sacreBLEU 30.54 for end-to-end T5-large, against 11.00 for the best pipeline tab. 4. The official repository releases the end-to-end training code, configs and data; the pipeline code is not released.

Problem

Table question answering datasets usually require short-form answers, such as an entity or an aggregate, so they under-test reasoning and integration over several facts §1 tab. 1. FeTaQA asks for a free-form answer, which the authors frame as generation over a table and its metadata: given a question q, a table T and table metadata m, produce an answer a §2.1. The answers are human-generated explanations, not spans copied from the source §1. The task demands that a system retrieve several cells, reason over their relations, integrate them into one coherent answer, and sometimes add explanation when the question is abstract §1.

Method

Dataset construction §2

1. Sample ToTTo tables with 3 to 34 rows and 3 to 7 columns, which is up to the 75th percentile of ToTTo table sizes, to avoid truncating linearised tables §2.2. Keep tables whose highlighted region spans several rows, plus a subset of single-row regions whose content spans several rows or columns §2.2. This gives 16,576 candidate instances §2.2.
2. Annotators write a question whose answer is the ToTTo sentence. They may edit the sentence, the table cell content and the highlighted region §2.2.1.
3. Question writing used 15 internal annotators and Amazon Mechanical Turk workers with at least 500 completed HITs, a 97% approval rate, and English-speaking locations §2.2.1.
4. Auto-reject questions whose length falls outside 4 to 25 (unit not stated) or that contain more than two interrogatives; this removed 259 examples §2.2.1.
5. Internal evaluators label each remaining annotation approve, reject or questionable, score it, and fix rejected questions §2.2.1.
6. Final data: 10,330 instances, 3,039 (30%) internal and 7,291 (70%) from Mechanical Turk §2.2.1.
7. Split by similarity so test instances are least similar to training. Compute the Jaccard similarity of token sets from table headers and questions. Seed with 800 random instances, add every instance with similarity above 0.465, which gives a 70/30 split. The 30% is divided 1:2 into dev and test §2.3.

The paper has no numbered equations or algorithm blocks. Its figures are described in words here and cited by number.

Pipeline model §3.1 fig. 4 fig. 5

Two modules, trained separately. Stage 1 predicts the highlighted cells with a semantic parser. Stage 2 writes the sentence from those cells.

# Pipeline model  [§3.1] [fig. 4]
# Input:  table (rows, title, page title); question
# Output: free-form answer (string)

def pipeline_answer(table, question):
    # Stage 1: TAPAS-base predicts denotations (highlighted cells)  [§3.1]
    cells = tapas_denotations(table, question)
    # Each predicted cell becomes a triple with its column header  [§3.1]
    triples = []
    for cell in cells:
        triples.append(["[TABLECONTEXT]", cell.header, cell.value])
    # Metadata enters as two more triples, using field names  [§3.1]
    triples.append(["[TABLECONTEXT]", "TABLE_TITLE", table.title])
    triples.append(["[TABLECONTEXT]", "PAGE_TITLE", table.page_title])
    # Stage 2: T5 data-to-text writes the answer  [§3.1] [§4.1]
    return data_to_text_t5(triples)

Stage 1 starts from a TAPAS-base checkpoint already fine-tuned on WikiTableQuestions and is further fine-tuned on FeTaQA, using the highlighted cells as weak supervision §3.1 fig. 5. The aggregation label is always NONE during this fine-tuning, so aggregation is left to stage 2 §3.1. The zero-shot variant skips this fine-tuning and uses the same checkpoint §4.1.

Stage 2 is a T5 model fine-tuned first on DART, then on 8K ToTTo instances that are not used for FeTaQA, to accept the triple format §3.1 §4.1. The paper calls this T5-large §4.1.

End-to-end model §3.2 fig. 6

A text-to-text model maps the linearised table and question to the answer.

# Table linearisation for the end-to-end model  [§3.2] [fig. 6]
# Input:  table_rows (list of rows of cell strings); question (string)
# Output: source string for the encoder

def linearise(table_rows, question):
    # Cells in a row are joined by spaces; rows are joined by [SEP]  [§3.2]
    row_texts = [" ".join(row) for row in table_rows]
    table_text = "[SEP]".join(row_texts)
    # The question comes first, each part introduced by [CLS]  [§3.2]
    source = "[CLS]" + question + "[CLS]" + table_text
    # T5 task prefix is added in preprocessing  [§4.1]
    return "summarize: " + source

# End-to-end fine-tuning  [§3.2] [§4.1]
# Input:  pretrained T5 model; train pairs (table, question, answer)
# Output: fine-tuned model

def train_end_to_end(model, tokenizer, train_pairs):
    # Add [CLS] and [SEP] as tokens and grow the embedding table  [§3.2]
    tokenizer.add_special_tokens(["[CLS]", "[SEP]"])
    model.resize_token_embeddings(len(tokenizer))
    for table, question, answer in train_pairs:
        # Source is truncated to 512 tokens  [code: end2end/train.py#L452]
        source = linearise(table, question)
        # Padding in the target is ignored by the loss  [code: end2end/Args.py#L165-L170]
        loss = model(source, labels=answer)
        # Update the weights (optimiser not stated in paper)  [§4.1]
        update(model, loss)
    return model

Hyperparameters

Results

Author-reported. Setup: the two benchmark methods, pipeline and end-to-end, scored on the test split tab. 4 §4.1. Metrics are n-gram and embedding-based comparisons of generated and reference answers §4.2.

Denotation retrieval, test split tab. 5. Scores compare predicted and reference highlighted cells.

The paper reports that the reference has a median of six highlighted cells, while the predicted set averages two cells before fine-tuning and three after §4.3.

Human evaluation: percentage of outputs scoring 4 or 5 on a 1 to 5 scale, over 100 samples, five internal evaluators §4.4 tab. 6.

Dataset statistics tab. 2: median and average question length 12 / 13.2 words, answer length 18 / 18.9 words, rows 12 / 13.8, columns 5 / 5.9, highlighted cells 6 / 8.0, cells highlighted 10.7% / 16.2%.

Annotation quality tab. 3: share of 1 to 5 scores of 4 or higher, percent agreement, and Randolph's kappa, over 100 samples. Question complexity 52.6%, 0.65, 0.48; denotation correctness 89.0%, 0.88, 0.82; denotation adequacy 91.6%, 0.89, 0.83; answer fluency 95.0%, 0.92, 0.89; answer correctness 92.4%, 0.91, 0.86; answer adequacy 90.6%, 0.88, 0.82; answer faithfulness 95.6%, 0.93, 0.89.

Paper vs code

Code: https://github.com/Yale-LILY/FeTaQA at commit bbc441be807212d736c8c35a18146b530dde11d0. Covers the end-to-end model and the released data. Its pipeline/ folder is empty and the top-level README says the TAPAS pipeline is "To be released..." README.md.

• Epochs. Paper: T5-small 30, T5-base and T5-large 80 §4.1. Configs: 60 for both small and large t5-small.json t5-large.json. No T5-base config is shipped.
• Batch size. Paper: 16 per device for small and 4 for large §4.1. Configs: 4 for small and 8 for large t5-small.json t5-large.json.
• Model parallelism. Paper: T5-large layers are split over 8 K80 GPUs §4.1. Code: the device map is commented out train.py. The README says training supports multiple GPUs via CUDA_VISIBLE_DEVICES README.md.
• ROUGE. Paper reports ROUGE-1, -2 and -L §4.2 tab. 4. Configs list meteor, bleu, sacrebleu, bertscore and bleurt, and no ROUGE t5-small.json t5-large.json. The argument help lists the implemented metrics without ROUGE Args.py, though the postprocessing has a ROUGE branch train.py.
• Target length. Paper: not stated. Configs: 60 tokens t5-small.json. The end2end README's example says 25 README.md.
• Input. Paper: linearises table and question only §3.2. Code matches: the linearisation takes the table and question, not the page or section titles utils.py.
• Step counts. The paper's step counts do not follow from its batch sizes. Our arithmetic, not the authors': T5-base, 80 epochs over 7,326 instances at 4 devices times 4 per device, gives 458 steps per epoch and 36,640 steps, matching the paper. T5-small, 30 epochs at 4 devices times 16 per device, gives about 3,430 steps, not about 6,900. The effective batch for small is therefore not stated.
• Dataset split. Paper's split procedure §2.3. The split code is not in the repository. The shipped jsonl files have the paper's counts: 7,326 train, 1,001 dev, 2,003 test fetaQA-v1_train.jsonl fetaQA-v1_dev.jsonl fetaQA-v1_test.jsonl.
• Predictions. end2end/outputs/ holds test predictions for small, base and large. Each file has 2,003 lines, the last without a trailing newline t5-large-test-predictions.txt. We did not recompute the metrics from these files.
• Pinned versions. Paper: none. Code: torch 1.7.1 and transformers 4.4.0 env.yml.

No discrepancies found for the linearisation, the summarize: prefix or the special-token setup, which match §3.2 §4.1.

Pitfalls

• Pipeline cannot be reproduced from this repository. Stage 1 (TAPAS) and stage 2 (T5 data-to-text) fine-tuning settings are not stated, and the code is not released README.md. The 8K ToTTo sample is not identified §3.1.
• Metric naming. Headline numbers are sacreBLEU, with the signature given in the Table 4 footnote tab. 4. The code's bleu option splits on spaces and is not sacreBLEU train.py. The sacreBLEU path is train.py.
• Metric models unnamed. BERTScore uses lang="en" and no model is named train.py. BLEURT is loaded with the library default and the checkpoint is not named train.py.
• Decoding unstated. The paper does not give a decoding strategy. The config leaves the beam count unset (None) Args.py, so the generation default applies. That default depends on the library version, which env.yml pins to transformers 4.4.0 env.yml.
• Silent truncation. Inputs longer than 512 tokens are truncated train.py. Table sampling was designed to avoid this §2.2, but the paper does not report how many instances exceed the limit. Targets are capped at 60 tokens in the configs t5-small.json. The paper does not state an answer cap and does not report the length of the longest answers.
• Table metadata is not an input. The dataset defines metadata m §2.1, and the pipeline uses the page and table titles §3.1. The end-to-end input does not include them utils.py. An implementer who expects the title to help the end-to-end model must add it, and the paper does not test that.
• Annotators could change the table. Annotators could edit cell content and highlighted regions §2.2.1, so FeTaQA tables and highlights may differ from the ToTTo source they came from.
• Unclear T5 size in human evaluation. Table 6 labels the source End-to-End without a size tab. 6. The text refers to the best end-to-end outputs §4.4 without naming the size.
• Human evaluation is small and internal. 100 samples, rated by five internal evaluators §4.4.
• Question length unit. The 4 to 25 filter does not state its unit §2.2.1.
• Re-splitting changes difficulty. The split is designed so that test items are least similar to training. The authors say a random split would put similar tables in several splits §2.3. Use the shipped split.
• Version. This card covers arXiv v1 (April 2021). The repository README cites a later TACL article README.md. Its text and numbers may differ, and they were not compiled here.
• Bottleneck. The authors suspect that poor denotation prediction and lost relations between denotations explain the pipeline's weak results §4.3. Fine-tuning TAPAS on FeTaQA raises the score only a little, from 9.16 to 11.00 sacreBLEU tab. 4.

Lineage

• Builds on: T5 for the text-to-text models §3.2 §4.1; TAPAS for the table semantic parser §3.1; ToTTo as the data source §2.2; DART for the stage 2 pretraining §3.1; WikiTableQuestions for the TAPAS starting checkpoint §4.1. None of these has a card in this wiki yet.
• Compared against: table QA datasets with short answers, including WikiTableQuestions, SequentialQA, WikiSQL, Spider and HybridQA §1 tab. 1.
• Superseded by: not known from wiki content. The repository README links a later TACL article; see Pitfalls.