TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

In plain words

Many useful datasets are tables: rows and columns in a database or on a web page. A language model like BERT learns from plain text, so it can read a question but does not understand how a table is laid out. TaBERT is a BERT-style model that is pretrained on tables together with the sentences written around them, so it learns how column names and cell values relate to ordinary words.

A table can have many rows, and reading all of them is too slow. TaBERT reads only a few rows that match the question, and a small extra attention step lets those rows share information about each column. The output is an encoder that you can plug into a system that turns questions into database queries. The authors report that such systems do better with it. All numbers in the paper are the authors' own.

TL;DR

TaBERT is a BERT-initialized encoder pretrained on about 26.6 million table-and-text pairs. It encodes a question together with K sampled rows, and vertical self-attention links those rows. Plugged into a WikiTableQuestions parser (MAPO) it gives 52.3% test accuracy for the run that was best on dev, above the previous ensemble's 46.9%. On Spider it is competitive with the leaderboard on dev. The released code covers the encoder and pretraining, not the parsers.

Problem

Semantic parsing turns a question into a query over a table. The model must understand the table's columns and cell values and align question words with them §1. Existing neural parsers attach BERT to a table in their own ways, and BERT was never pretrained on structured data §1. A table can have many rows, and encoding all of them with a large Transformer is too expensive §1 §3.1. The paper targets both structured SQL databases (Spider) and semi-structured Web tables (WikiTableQuestions) §4.

Method

Inputs. For an utterance and a table, TaBERT first builds a content snapshot of K rows §3.1. For K = 1 the snapshot is one synthetic row that takes, from each column, the cell with the highest n-gram overlap with the utterance. For K > 1 it takes the top-K rows by n-gram overlap, with n at most 3 §3.1.

Row linearization. Each snapshot row becomes one input sequence: the utterance tokens, then one string per cell. A cell string holds the column name, a data type (text or real), and the cell value, separated by a vertical bar. Cell strings are joined with [SEP] §3.1 eq. 1. The data type is predicted by majority vote over the spaCy NER labels of the cell tokens §3.1 app. A.1.

Encoding. A BERT model encodes each row sequence separately §3.1. Each cell's vector is the mean of the BERT outputs over its tokens §3.1.

Vertical self-attention. For K = 3, V = 3 vertical layers then run over the aligned vectors: each utterance token or cell attends across the rows at its own position. Each layer has the same form as a Transformer layer §3.1 §5.

Outputs. A column vector is the mean of that column's cell vectors over the snapshot rows. An utterance-token vector is the mean over rows in the same way §3.1. The paper also describes an optional table vector from the [CLS] position §3.1.

Pretraining data. Training pairs come from English Wikipedia tables with their three preceding paragraphs, and from the WDC WebTable Corpus (2015, relational subset). After heuristic cleaning this gives 1.3M Wikipedia tables and 25.3M WDC tables, 26.6M in total §3.2 app. A.1.

Pretraining instances. Non-overlapping sliding windows of at most 128 tokens over the context give the utterance. K rows are sampled at random from the table to form the snapshot §5.

Objectives.

• Masked language modelling on the context, 15% of sub-tokens §3.2.
• Masked Column Prediction: 20% of columns have their names and types masked in every row; the model predicts the masked tokens from the column vector §3.2.
• Cell Value Recovery: for the masked columns, the model predicts each cell's value tokens from its cell vector with a span-based head: a two-layer GeLU network over the cell vector and a position embedding §3.2. Cell values are not masked in the input §3.2 [fn. in §3.2].
• The training loss is the sum of the three objectives app. A.2.

Initialization and variants. TaBERT-Base and TaBERT-Large start from uncased BERT-Base or BERT-Large §5. K = 1 has no vertical layers and no CVR objective §5. K = 3 uses three vertical layers §5.

Parsers. For Spider, the encoder feeds a TranX-based parser that generates SemQL, fine-tuned jointly with the encoder §4.1 app. B.1. For WikiTableQuestions, the LSTM encoder of MAPO is replaced by TaBERT §4.2 app. B.2.

# 1. Select K rows most relevant to the utterance  [§3.1]
snapshot = top_k_rows_by_ngram_overlap(utterance, table, k=3)

# 2. Build one sequence per row: utterance, then cell strings  [§3.1] [eq. 1]
for row in snapshot:
    # cell string: "column | type | value", cells joined by [SEP]
    cells = [cell_string(column, row[column]) for column in table]
    sequence = [CLS] + utterance_tokens + [SEP] + join(cells, [SEP])

    # 3. Encode the row with BERT  [§3.1]
    token_vectors = bert(sequence)              # (seq_len, d)

    # 4. Mean-pool each cell's tokens into one vector  [§3.1]
    cell_vectors[row, col] = mean(token_vectors[span_of(col)])

# 5. Vertical self-attention across rows, per position  [§3.1] [§5]
for layer in vertical_layers:                   # V = 3
    hidden = layer(hidden)                      # attends over the row axis

# 6. Pool over rows: columns and utterance tokens  [§3.1]
column_vectors = mean(hidden[:, cell_positions], axis=rows)
token_vectors = mean(hidden[:, utterance_positions], axis=rows)

# 7. Pretraining loss: plain sum of the three objectives  [§3.2] [app. A.2]
loss = mlm_loss + mcp_loss + cvr_loss

Hyperparameters

Results

Author-reported. The setup is WikiTableQuestions (execution accuracy) with a MAPO parser §4.2 §5, and Spider (exact-match accuracy on the public dev set) with a TranX-based parser §4.1 §5. Hardware is in app. A.2.

WikiTableQuestions tab. 1. "Test→Best" is the test score of the run that was best on dev, as the caption defines it.

The authors' 10-run means and standard deviations are in the table tab. 1. The text quotes 52.3% as the single-model test result §5.

Spider tab. 2. The table is labelled as the public development set, not the test set. Mean ± std over 5 runs, with the best run alongside.

Ablations on WikiTableQuestions dev tab. 4 (TaBERT-Base, K = 1 pretraining): column name only 49.6 (WTQ) and 60.0 (Spider); column name and type without cell values 49.9 and 60.4; column name, type and cell value 51.2 and 63.3. BERT with the column name only: 49.0 and 58.6. BERT with the TabFact "column is value" template: 50.2 and 63.1 tab. 4.

Objectives tab. 5 (TaBERT-Base, K = 3, mean ± std): MCP only gives 51.6 ± 0.7 on WTQ dev and 62.6 ± 0.7 on Spider dev; MCP + CVR gives 51.6 ± 0.5 and 63.3 ± 0.7. The authors call the CVR advantage slight §5.

Snapshot example tab. 3: qualitative only, one WTQ dev question, showing that K = 3 keeps the two rows needed for a subtraction question, while K = 1 does not.

Paper vs code

Code: https://github.com/facebookresearch/TaBERT, commit 74aa4a88783825e71b71d1d0fdbc6b338047eea9, license CC BY-NC 4.0 (LICENSE.md). Non-commercial only. This card paraphrases the code and cites it, and quotes nothing from it.

Search trail. The paper's footnote links fburl.com/TaBERT. That link redirects to an internal Meta address, which I did not follow. A GitHub search for TaBERT in the facebookresearch organisation returned facebookresearch/TaBERT. Its README cites this paper, and the authors were at Facebook AI Research. I cloned it at the commit above.

What the repo contains. The encoder, the pretraining data scripts and the pretraining loop. It does not contain the Spider parser (TranX-based) or the WTQ parser (MAPO-based); Appendix B cannot be checked against code. The released encode() returns only context and column encodings vertical_attention_table_bert.py.

Discrepancies:

• Snapshot at inference. The paper selects rows by n-gram overlap, n ≤ 3, and builds a synthetic K = 1 row §3.1. The released vertical encoder takes the first K rows of the table input_formatter.py. Pretraining samples K rows at random input_formatter.py. The K = 1 path reads a sample_value_tokens field set by the caller input_formatter.py. Pretraining sets one random cell per column input_formatter.py. No n-gram selection exists in the repo.
• Column masking loss. The paper says the MCP objective predicts the bag of masked tokens with a multi-label classifier §3.2. The code takes one cross-entropy term per masked token, each from the column vector of that token vertical_attention_table_bert.py input_formatter.py.
• Losses are sums over tokens. Each term is a sum over masked tokens, not a mean vertical_attention_table_bert.py. The paper also says "sum" app. A.2, but a sum weights each objective by its number of masked tokens.
• Column representation for K = 1. The paper says column vectors are the mean over column-name tokens §5. The released default mean_pool pools over the whole cell string vanilla_table_bert.py. The released K = 1 command does not set column_representation README.md.
• Cell Value Recovery default is off. The config default is predict_cell_tokens=False config.py. The released K = 3 command turns it on README.md. The paper's K = 3 results include the CVR objective §5 tab. 5.
• Weight decay for K = 1. The paper's table says 0.01 for all variants tab. 6. The released K = 1 command uses 0.0 README.md.
• Context window. The paper says 128 tokens §5. The released data command uses 128 README.md. The config default is 256 config.py.
• Vertical head count. The paper does not state it. The code defaults to 6 config.py and the K = 3 command uses 6 README.md.
• [CLS] table vector. The paper describes an optional table vector §3.1. The released forward pass returns only context and column encodings vertical_attention_table_bert.py.
• New vertical layers. The paper does not say how they are initialized. The code applies BERT's weight initialization to the new modules vertical_attention_table_bert.py.
• Learning-rate schedule. Not stated in the paper. The code uses polynomial decay and a warmup of 10% of updates train.py train.py.
• Masking rule. The 80/10/10 replacement rule is not in the paper. The code uses it input_formatter.py.

Pitfalls

• The README's Large K = 3 setup fails as written. The vertical attention requires the hidden size to be divisible by the head count vertical_attention_table_bert.py. BERT-Large has hidden size 1024 config.py, and 1024 is not divisible by 6. The paper does not give the head count for either size, so an implementer must choose a divisor of the hidden size and state the choice.
• Snapshot selection is not released. The paper's n-gram selection §3.1 must be written from the description. The paper says the heuristic picked the best row in 40 of 50 sampled WTQ examples §3.1, which is the authors' own check, not a benchmark.
• Train and test mismatch. Pretraining samples K rows at random, and K = 1 uses random cells input_formatter.py. Parsing uses similarity-selected rows §3.1. The repo does not measure the effect.
• Vertical attention has no row-index term. The vertical layers contain no row-position embedding vertical_attention_table_bert.py, so the rows of a snapshot are treated as an unordered set.
• The attention mask is additive with a large negative constant. Padded positions get −10000 added to their scores, not −∞ vertical_attention_table_bert.py. Training uses --fp16 README.md.
• Cost scales with K. Each snapshot row runs through BERT separately vertical_attention_table_bert.py, so a K = 3 model costs about three encoder passes per table. Compute for the largest model is about six days on 128 V100 GPUs app. A.2.
• Data types need spaCy. The real/text label comes from a spaCy NER majority vote §3.1 app. A.1. Pretraining data cannot be rebuilt from this repo alone: the scripts need Wikipedia and Common Crawl input, and the 26.6M corpus is not included.
• Licensing. The repo is CC BY-NC 4.0 README.md LICENSE.md. The README links pretrained weights on a Google Drive folder README.md and does not state a separate licence for them. Do not use the code or weights commercially until that is resolved.
• Test-set overlap. The authors say some of the 421 WTQ test tables may be in the pretraining corpus, but judge the share tiny §5 footnote. They also report that Wikipedia-only pretraining did worse §5 footnote.
• Quote the right WTQ number. 52.3% is the Test→Best value; the mean is 51.8 ± 0.6 tab. 1.
• Spider numbers are dev numbers tab. 2.
• The table leaves out details that matter. No warmup, Adam epsilon, learning-rate schedule, or masked-token cap is in the paper. The repo sets them train.py README.md README.md config.py.

Lineage

• Builds on BERT (Devlin et al., 2019) §1 §3.1 and the masked language modeling objective §2.
• Parsers used: TranX §4.1 with the SemQL grammar from IRNet §4.1; MAPO §4.2 (Liang et al., 2018).
• Compared against: the Spider leaderboard entries in tab. 2 and the WTQ entries in tab. 1.