PubTables-1M: Towards comprehensive table extraction from unstructured documents

In plain words

A table on a scientific page is easy to read for a person, but a computer sees only text and lines. To learn to find tables and understand their rows, columns and headers, a model needs many examples with exact answers. The authors built such a set of examples from nearly one million tables in open-access papers.

The labels come from the paper's markup, joined to where the text sits on the page. The authors found that markup often splits one header cell into several blank cells. That leaves the same table with two valid readings, which confuses both training and scoring. They merge those cells back into one cell, then drop tables whose text does not match the page.

They then train a general-purpose detector, DETR, on the three tasks: finding the table, finding its rows and columns, and linking headers to the cells below them. It works on all three without special changes. The numbers are the authors' own.

TL;DR

• A dataset of 947,642 tables from PubMed Central, with word-level alignment to PDF text and boxes for tables, rows, columns, cells and headers §3.
• Canonicalization merges cells that markup oversegments, using header rules tied to the PMCOA markup §3, alg. 1.
• DETR is applied unchanged to table detection, structure recognition and functional analysis §4.
• Author-reported: on all test tables, DETR reaches 0.8138 content accuracy and 0.9845 GriTS_Top; Faster R-CNN on the same data reaches 0.1039 and 0.8616 tab. 4.

Problem

Crowd-sourced table datasets built from markup (HTML, XML, LaTeX) have three gaps §1, §2. Markup gives no cell coordinates, so the logical structure is only implicit §1. Column headers are often incomplete and row headers are not annotated at all §3. Markup also oversegments: a header cell that visually spans several columns or rows is stored as several grid cells, often with blank cells added §1, fig. 3.

The authors argue that this breaks supervised learning. An oversegmented annotation contradicts the table's logical reading, so the same table has two valid ground truths. This violates the usual assumption of one correct answer per table, which produces contradictory training signal and underestimates test performance §1. Their survey of prior sets (ICDAR-2013 at 248 tables for detection and structure, TableBank, SciTSR, PubTabNet, FinTabNet) finds that none gives bounding boxes for all cells including blank ones, for rows and columns, and for row headers together §2, tab. 1. About 13.9% of tables in PubTabNet and 36.2% in FinTabNet, out of all tables investigated, have an oversegmented projected row header; PubTables-1M has none by its own rule tab. 2.

Method

Source and scope. Tables come from the PMCOA corpus: each article is a PDF and an XML document, and each table's content and structure is marked up with HTML tags §3, first paragraph. The PDF and XML have no given correspondence and no table positions §3. Tables that span pages are out of scope [§3, stats paragraph].

Step 1: Alignment §3, Alignment. The PDF is turned into a character sequence, each character with a box. The authors align it with the text of each table's HTML using the Needleman-Wunsch algorithm §3, Alignment. Each cell with text gets a text cell box: the union of its characters' boxes §3, Alignment.

Step 2: Completion §3, Completion. Boxes are then completed for rows, columns, the table and grid cells, as follows.

• The table box is the union of all text-cell boxes.
• A row spans the full table width. Its vertical extent is the union of the text cells that start or end in that row.
• A column spans the full table height. Its horizontal extent is the union of the text cells that start or end in that column.
• A grid cell is the intersection of the union of its rows with the union of its columns. It is defined even for blank cells, unlike the text-cell box [§3, Completion].

# Complete row, column and grid-cell boxes  [§3, Completion]
# Input:  cells with optional text_box; each cell knows its row and column span
# Output: a box for every row, every column and every cell's grid cell

def complete_boxes(cells, rows, columns):
    # The table box is the union of every text-cell box  [§3, Completion]
    table_box = union([cell.text_box for cell in cells if cell.text_box])
    for row in rows:
        # Rows span the full table width  [§3, Completion]
        row.box = full_width(table_box, union_y_of_cells_touching(row, cells))
    for column in columns:
        # Columns span the full table height  [§3, Completion]
        column.box = full_height(
            table_box,
            union_x_of_cells_touching(column, cells),
        )
    for cell in cells:
        # Grid cell: rows union intersected with columns union  [§3, Completion]
        cell.grid_box = intersect(
            union([row.box for row in rows_of(cell)]),
            union([column.box for column in columns_of(cell)]),
        )

Step 3: Canonicalization §3, Canonicalization; alg. 1. The goal is to correct oversegmentation so that each table has one structure reading. It assumes the Wang model of table structure: each header is a tree, and each internal node has at least two children §3, Canonicalization. The authors report that the Wang model holds for 97% of tables they observed [§3, Canonicalization]. Each body column maps to a unique leaf of the column-header tree, and each body row to a unique leaf of the row-header tree §3.

The algorithm has two blocks: first it adds cells to the column and row headers, then it merges cells alg. 1.

# Canonicalize one table's structure annotation  [alg. 1]
# Input:  cells of one table (row span, column span, text, header flag)
# Output: the same cells with merged spans and header labels

def canonicalize(table):
    # Split every blank spanning cell into blank grid cells  [alg. 1]
    split_blank_spanning_cells(table)
    # If row 0 starts with a blank cell, it is column header  [alg. 1]
    if first_cell_of_row_0_is_blank(table):
        mark_column_header_row(table, row=0)
    # Add rows to the column header until every column has a complete
    # cell that spans only that column  [alg. 1]
    if has_column_header_row(table):
        while not every_column_has_own_complete_cell(table):
            mark_next_row_as_column_header(table)
    # A body row with exactly one non-blank cell, in the first column,
    # is a projected row header  [alg. 1]  [§3, Canonicalization]
    for row in body_rows(table):
        if is_projected_row_header(row):
            mark_projected_row_header(row)
    # If the first body column has a spanning or blank cell, that
    # column joins the row header  [alg. 1]
    if first_body_column_has_spanning_or_blank_cell(table):
        mark_first_body_column_as_row_header(table)
    # Merge column-header cells with adjacent cells above and below
    # that span exactly the same columns, recursively  [alg. 1]
    for cell in column_header_cells(table):
        merge_same_span_neighbours_above_and_below(cell)
    # Merge a column-header cell with adjacent blank cells below it,
    # if every cell below it is blank and in the column header  [alg. 1]
    for cell in column_header_cells(table):
        merge_blank_cells_below_if_all_blank(cell)
    # Merge a column-header cell with adjacent blank cells above it,
    # if every cell above it is blank  [alg. 1]
    for cell in column_header_cells(table):
        merge_blank_cells_above_if_all_blank(cell)
    # Merge all cells of a projected row header into one cell  [alg. 1]
    for row in projected_row_headers(table):
        merge_all_cells_in_row(row)
    # Merge each row-header cell with adjacent blank cells below it  [alg. 1]
    for cell in row_header_cells(table):
        merge_blank_cells_below(cell)

Three cases need conventions §3, Canonicalization:

• Blank rows between a parent header cell and its children: the blanks join the child below, not the parent above §3, Canonicalization.
• A blank stub head in the top-left corner: its blank cells are merged per column, consistent with the column-header merge §3, Canonicalization.
• The authors assume oversegmentation happens only where it does not change how the table looks: not along the direction of centred text, usually vertically in headers, and horizontally in projected row headers §3, Canonicalization.

Row headers are only partly inferred. Only projected row headers and first-column cells are added, because full row-header inference is out of scope [§3, Canonicalization].

Step 4: Quality control §3, Quality control. The authors filter tables automatically, because manual checking is impractical. Each check and its threshold is listed in the hyperparameter table. Tables are dropped when they have overlapping rows or columns, when text differs too much between XML and PDF, or when words do not sit in cells. The text check compares XML text with the PDF text inside each grid cell, and PDF text counts as ground truth where they differ §3, Quality control. Tables with more than 100 objects are removed as outliers; the authors report that under 0.1% of tables are discarded this way §3, Quality control.

Step 5: Detection data and splits [§3, Dataset statistics and splits; app. Training Details]. Detection training uses pages in which every table passed quality control, so that no page has a missing table annotation [§3, Quality control; code: scripts/process_pubmed.py#L1648-L1652]. Splits are random, at the document level, 80/10/10 §3, Dataset statistics and splits.

Step 6: Detection model §4, first paragraph. Table detection has two classes: table and table rotated. A rotated table is turned 90 degrees counterclockwise §4. Training renders pages to images with a maximum side of 1000 pixels and scales the boxes to match app. Training Details.

Step 7: Structure and functional-analysis model [§4, TSR and FA model; fig. 4]. One detector predicts six object types: the whole table, its columns, its rows, its column-header cells, its projected row headers and its spanning cells §4. The intersection of each column and row is an implicit seventh type, the grid cell §4. Training uses dilated boxes, which fill the gaps between neighbouring rows and between neighbouring columns: each shared boundary moves to the midpoint of the gap, and objects follow their rows and columns §4. Inputs are cropped to the table with 30 pixels of padding on each side, or fewer where the page ends app. Training Details.

Step 8: Inference app. Inference. Objects are parent–child linked by containment. No object may have two parents of the same class. A conflict exists when two parents of one class both overlap a child by 50% of its area. The ground truth is conflict-free, so the authors add a conflict-resolution step at inference: suppress objects or adjust their boxes, similar to non-maximum suppression but with overlap measured in children app. Inference. Objects are then converted to a logical table app. Inference.

Step 9: Scoring app. Scoring. After cell text is extracted, the row boxes' vertical extents and the column boxes' horizontal extents are tightened to the text they contain. Cell content does not change. Location scores use the tightened cells, because the dilated training boxes would otherwise be penalised as padding app. Scoring.

Hyperparameters

Values are from the paper unless the citation says code. Where the paper says "default", the code value is listed.

Results

Author-reported. Setup: PubTables-1M test split; DETR and Faster R-CNN on ResNet-18; TSR and FA evaluated with GriTS and content accuracy [§5, tab. 3, tab. 4]. The paper does not report the number of runs or variance.

Dataset §3, Dataset statistics and splits; tab. 1.

• 947,642 tables for structure recognition, of which 52.7% are complex (at least one spanning cell). Before canonicalization, 40.1% were complex by the original annotation. Canonicalization changes 34.7% of tables, or 65.8% of complex ones.
• Structure split: 758,849 train, 94,959 validation, 93,834 test.
• Detection split: 460,589 train, 57,591 validation, 57,125 test pages.
• Oversegmentation survey of projected row headers tab. 2: PubTables-1M has 153,705 tables with a projected row header and 0 oversegmented. Comparison rows: SciTSR 54 of 342 (15.79%), PubTabNet 58,747 of 100,159 (58.65%), FinTabNet 25,348 of 25,637 (98.87%).

Table detection tab. 3. Test performance:

Table structure recognition tab. 4. Metrics: Acc_Cont (share of tables whose cell text matches exactly), GriTS_Top, GriTS_Cont, GriTS_Loc and Adj_Cont (adjacency F-score) §5. "Canonical" and "Non-canonical" name the test data.

Claims the table supports §5:

• On canonical test data, DETR is best on every metric. The paper says canonical training data improves complex-table accuracy from 0.5360 to 0.6944 §5. The 0.5360 is DETR-NC scored on non-canonical test data, and 0.6944 is DETR scored on canonical test data, so the two numbers do not share a test set tab. 4.
• On simple tables, DETR-NC scores higher on canonical test data (0.9349) than on non-canonical data (0.8678). The paper uses this to argue that canonical data is less noisy and gives a more reliable evaluation §5.

The GriTS metric is defined in eq. 1:

\mathrm{GriTS}_f(\mathbf{A}, \mathbf{B}) = \frac{2 \cdot \sum_{i,j} f(\tilde{\mathbf{A}}_{i,j}, \tilde{\mathbf{B}}_{i,j})}{|\mathbf{A}| + |\mathbf{B}|} eq. 1

A and B are the ground-truth and predicted tables as matrices. Tilde marks the most similar substructures, which are selections of m rows and n columns §5. f sets the view: topology, content or location §5. The paper says it is an F-score §5.

Paper vs code

Checked against the official repository, microsoft/table-transformer, at commit 16d124f616109746b7785f03085100f1f6247575 [code_commit]. The paper is the arXiv v3 version. The repository contains the PMCOA processing script, the padding script, the training and inference code and the two training configs. It does not contain the dataset files, the split code, the Faster R-CNN baseline code or code for the oversegmentation survey in tab. 2.

• Canonicalization rules differ from Algorithm 1. The code has these differences from alg. 1:
  • Header rows: the code makes row 0 a header when it has a column span over one column, and extends headers until a row has as many cell slots as there are columns process_pubmed.py. Algorithm 1 uses the blank-first-cell rule and the complete-cell rule alg. 1.
  • Projected row headers: the code labels a non-header cell that spans all columns process_pubmed.py. Algorithm 1 labels a row with exactly one non-blank cell in the first column, before merging [alg. 1, line for projected row headers].
  • Blank split and column-header merges: I did not find the "split blank spanning cells" step or the column-header merge steps in standardize_and_fix_xml_annotation process_pubmed.py. The code absorbs blank first-column cells into the content cell above them, and absorbs blank cells in projected row headers [code: scripts/process_pubmed.py#L819-L846], [code: scripts/process_pubmed.py#L855-L887].
• Quality-control edit distance. The paper averages the normalised edit distance over every cell §3, Quality control. The code pools all cells: distance is 2 * ED / (len_xml + len_pdf) summed over cells, with all whitespace removed process_pubmed.py. The threshold 0.05 matches process_pubmed.py.
• Quality-control filters not in the code. I searched scripts/ for the word-overlap threshold of 0.9, the 100-object outlier limit and the overlapping row or column test. None was found. The released pipeline may therefore not reproduce the paper's filtering exactly.
• Completion boxes. The paper uses the union over all cells that start or end in a row or column §3, Completion. The code takes the minimum start over cells that start in the column and the maximum end over cells that end in it, separately process_pubmed.py. Row boxes follow the paper's rule with the same split process_pubmed.py.
• Dilated boxes are not in the release. The paper's dilation step §4 has call sites in inference that are commented out. They refer to fill_row_gaps and fill_column_gaps, which are not defined in the repository [code: src/inference.py#L436-L440]. The GriTS_RawLoc docstring says the training boxes are dilated eval.py. The dilation code for training is not in this repository.
• Tightening for scoring is not found. The appendix describes tightening rows vertically and columns horizontally to the text app. Scoring. I did not find that step. align_rows and align_columns only snap rows and columns to the table box postprocess.py. The GriTS_Loc comparison in the paper therefore cannot be checked against the code.
• Augmentation. The paper says only "standard" augmentations such as random cropping and resizing app. Training Details. The code adds random erasing with p = 0.5 for structure training table_datasets.py. The crop sizes match the paper's 30 px, with an extra 10 px option [code: src/table_datasets.py#L467-L468].
• No-object weight for detection. The paper gives 0.4 for the TSR and FA model only app. Training Details. The detection config also sets 0.4 detection_config.json.
• Padding for dataset construction. The paper uses 30 px [app. Training Details]. The PMCOA script default is 30 px process_pubmed.py. The separate padding script defaults to 2 px [code: scripts/create_padded_dataset.py#L117]. Which padding produced the released data is not stated.
• Object classes. The code has the six structure classes and the two detection classes that the paper describes process_pubmed.py, main.py. The rotated-table class in code is driven by a portrait test on the table box process_pubmed.py.
• Matches. The paper's 6 and 2 classes, 125 and 15 queries, 20 epochs, the TSR + FA learning rate of 5e-5, the 0.4 no-object weight and step decay (drop 1, gamma 0.9) all match the configs structure_config.json, detection_config.json.
• Licence of the code. The root LICENSE is MIT. The detr/ subfolder has its own LICENSE, which is Apache-2.0 LICENSE. Code snippets from detr/ would fall under Apache-2.0. None are quoted here.

Pitfalls

• Canonicalization is PMCOA-specific. The paper says the algorithm was designed for the PMCOA markup and may not transfer to other sources [§3, Limitations]. Canonicalization does not guarantee error-free output [§3, Limitations]. The code's rules differ from alg. 1 (see Paper vs code), so an implementation must choose one reading and say which.
• Canonical and non-canonical scores are not comparable. Changing the test set changes every metric. Compare only models on the same test data, as [tab. 4] does.
• Single run, no variance. The paper gives one number per cell and no seeds or confidence intervals §5, tab. 3, tab. 4. The Faster R-CNN baseline has no stated training details beyond "default" and no code in the repository [§4, §5].
• Learning rate for detection is not clearly stated. The paper says the TD learning rate was chosen from three values on validation after one epoch app. Training Details, but does not say which value was chosen. The config has 5e-5 detection_config.json.
• Section and figure numbers. Section, table, figure and appendix numbers here follow the LaTeX \section, \begin{table} and \begin{figure} order in the arXiv v3 source. They were not checked against the PDF, and the appendix subsections are cited by name. If the compiled PDF numbers differ, fix the citations.
• Dilated boxes need extra code at inference. Training uses dilated boxes, so predicted boxes contain padding. The paper's inference untangles these via the conflict-resolution step app. Inference, but the dilation and tightening code are not in the repository (see Paper vs code).
• Quality filters may not match the release. The 0.9 word-overlap and 100-object filters are described in the paper but not found in the code. A pipeline built from the code alone may keep tables the paper removes.
• Oversegmentation survey code is missing. The projected-row-header counts in tab. 2 came from a survey the repository does not contain. The table reports "estimated" values tab. 2 caption and excludes tables under five rows.
• Images are not used here. The paper's figures show the example table, three tasks, the oversegmented header, and the object classes. They are described in words and cited as fig. 1 to fig. 4 only.

Lineage

• Builds on: DETR, applied without change to all three tasks §4; the Needleman-Wunsch alignment §3, Alignment; the Wang model of table structure §3, Canonicalization; GriTS for evaluation §5. The paper gives these as bibliography keys, not arXiv ids.
• Compared against: Faster R-CNN on the same data §5; the crowd-sourced datasets TableBank, SciTSR, PubTabNet and FinTabNet §2, tab. 1.
• Superseded: not known from this wiki's content.