Tabular Data: Deep Learning is Not All You Need

In plain words

Spreadsheet-style data, with one row per record and one column per feature, is the most common kind of data in industry. For years the go-to tool has been a family of tree-based models such as XGBoost. Several research groups have proposed neural networks built for this kind of data and claimed they beat the trees. Each of those papers tested on its own choice of datasets and tuned its own models, so it was hard to know who was right.

The authors ran a fair test. They took four of the proposed networks and XGBoost, trained all of them on eleven datasets, and tuned each one with the same search budget. XGBoost was usually the most accurate single model, and it was much quicker to tune. The best result came from combining the neural networks with XGBoost in one ensemble.

The practical message is that a tree model is a strong default for tabular data, and that adding neural networks to it can help. The authors caution that eleven datasets cannot cover all tabular problems.

TL;DR

• Compares four deep models for tabular data (TabNet, NODE, DNF-Net, 1D-CNN) with XGBoost on 11 datasets, using one tuning protocol §1 §3.1.
• XGBoost usually outperformed the deep models on datasets that were not in the papers that proposed them §3.2 tab. 2 tab. 3.
• XGBoost needed fewer hyperparameter-search iterations to plateau §3.2 fig. 2.
• An ensemble of the deep models and XGBoost gave the best mean relative result, 2.32% (author-reported) tab. 3.

Problem

Deep networks were proposed for tabular data and some were reported to beat gradient-boosted decision trees §1. Those comparisons used different datasets and different amounts of tuning, and some models lacked open-source code, so the claims could not be checked against each other §1. The paper asks two questions: whether the recent deep models are more accurate, especially on datasets that were not in their original papers, and how long training and hyperparameter search take compared with XGBoost §1.

The authors state that no single model is best for every problem (the "no free lunch" principle) and that their dataset selection cannot cover all tabular data §1.

Method

This is a benchmark study, not a new model. The four deep models are taken from their original papers and evaluated under a common protocol. Steps as the paper performs them:

1. Models. Four deep models are compared: TabNet, NODE, DNF-Net and 1D-CNN. Each is described in §2.1 with its key idea (sparse feature masks for TabNet, differentiable oblivious trees for NODE, soft DNF blocks for DNF-Net, and a 1D convolutional stack for 1D-CNN) §2.1. XGBoost is the baseline §2 §3.2.
2. Datasets. Eleven datasets are used. Nine come from the TabNet, DNF-Net and NODE papers, and two are Kaggle datasets that no earlier paper used §3.1.1 tab. 1. Each dataset is split as in its original paper §3.1.1 app. B.
3. Standardization. Each feature is standardized to zero mean and unit variance, with statistics from the training split only §3.1.1.
4. Tuning. Each model is tuned with Hyperopt's Bayesian optimization for 1,000 steps per dataset. The selection criterion is the smallest validation loss. The starting values come from the original paper, and the search spaces are in app. B §3.1.2.
5. Training. Deep models use Adam with no learning-rate schedule, following the original implementations. The batch size is tuned per dataset. Training stops after 100 consecutive epochs without validation improvement §3.1.2 app. B.
6. Evaluation. Each tuned configuration is run with four seeds, and test results are reported as mean ± standard error of the mean §3.1.2 tab. 2. Classification is scored with cross-entropy and regression with squared error, as the caption states (see Pitfalls) §3.1.2 tab. 2.
7. Significance. Friedman's test at the 95% level compares the models §3.1.2 §3.2.
8. Ensemble. Five models are combined: TabNet, NODE, DNF-Net, 1D-CNN and XGBoost §2.2. Two combination rules are given, a uniform mixture (eq. 1) and a weighted average whose weights come from validation loss (eq. 2) §2.2 eq. 1 eq. 2.
9. Ensemble subsets. Subsets of members are chosen by validation loss, by per-example uncertainty, or at random. Choosing by validation loss worked best on Shrutime §3.2 fig. 1.
10. Cost. Compute cost is measured by the number of hyperparameter-search iterations until performance plateaus, because the authors judge wall-clock time unfair across software implementations §3.2 fig. 2.

The combination rules as written:

p(y|x) = \sum_{k=1}^{K} p_{\theta_m}(y|x, \theta_m) eq. 1

p(y|x) = \sum_{k=1}^{K} l^{\text{val}}_k \, p_{\theta_m}(y|x, \theta_m) eq. 2

Here l^{\text{val}}_k is the validation loss of the k-th model §2.2. The text calls these weights the "normalized validation loss", but the formula does not normalize them (see Pitfalls) §2.2.

The subset procedure in pseudocode. The paper does not give an algorithm environment, so this follows the text §3.2:

# Ensemble prediction, validation-loss weighted  [§2.2] [eq. 2]
# Input:  models (K trained models), x, val_losses (K,)
# Output: class probabilities p(y|x)

def ensemble_predict(models, x, val_losses):
    # Weight each model by its validation loss, as eq. 2 is written
    weights = val_losses
    # Weighted sum of the per-model predictive distributions
    probs = sum(
        weights[k] * models[k].predict_proba(x)
        for k in range(len(models))
    )
    return probs

# Subset ordering by validation loss  [§3.2] [fig. 1]
# Input:  models (K trained models), val_losses (K,)
# Output: models ordered best first

def order_by_validation_loss(models, val_losses):
    # Lower validation loss comes first; this ordering performed best [§3.2]
    order = sorted(range(len(models)), key=lambda k: val_losses[k])
    return [models[k] for k in order]

Hyperparameter search, one dataset and one model §3.1.2 app. B:

# Tuning loop, one model on one dataset  [§3.1.2] [app. B]
# Input:  train, valid splits; search space
# Output: configuration with the smallest validation loss

def tune(train, valid, space):
    # Start from the configuration of the original paper  [§3.1.2]
    best_config = initial_config_from_original_paper()
    best_loss = evaluate(best_config, train, valid)
    # Bayesian search for 1,000 steps  [§3.1.2]
    for step in range(1000):
        # Propose a configuration from the search space  [app. B]
        config = propose(space, history)
        # Stop after 100 epochs with no validation improvement  [§3.1.2]
        loss = evaluate(config, train, valid, patience=100)
        if loss < best_loss:
            best_loss = loss
            best_config = config
    return best_config

Hyperparameters

Fixed settings and split rules. Search spaces for each model follow in the next table.

Search spaces, as written in the appendix. "Log-uniform [a, b]" is a log-scale uniform draw, and "Discrete uniform" draws integers app. B:

Results

Author-reported. Setup: 11 tabular datasets tab. 1, each model tuned with the same protocol, four seeds, test results as mean ± SEM §3.1 §3.2. Hardware and wall-clock times: not stated. Lower is better in tab. 2 and tab. 3.

• Mean relative deterioration (lower is better): the deep-model ensemble with XGBoost scored 2.32%. XGBoost scored 3.34%, the simple classical ensemble 3.15%, and the single deep models 7.56% (1D-CNN) to 14.21% (NODE). The ensemble without XGBoost scored 6.91% tab. 3.
• Unseen datasets: XGBoost outperformed the deep models on 8 of the 11 datasets that the deep models' own papers did not use, with p<0.005 §3.2.
• Ensemble subsets: of the three subset orderings, averaging by validation loss was best, and three models were near-optimal on Shrutime. Random ordering was the worst §3.2 fig. 1.
• Search cost: on Shrutime, XGBoost reached good performance in fewer hyperparameter-search iterations than the deep models. The authors report XGBoost training as more than an order of magnitude faster, without figures §3.2 fig. 2.
• Replication claim: the authors say their results on each model's original datasets matched the original papers §3.2.

Per-dataset results (mean ± SEM, test set). Bold marks the best entry per row, as in the paper. Rossmann and YearPrediction are reported as MSE, the other rows as cross-entropy × 100 tab. 2:

Average relative performance on each model's unseen datasets, geometric mean tab. 3 (lower is better):

Datasets tab. 1:

Paper vs code

No official code. Searched: the arXiv abstract page (no code link); GitHub repository search for the title and key phrases, for the authors' names, and for the authors' personal account (its repositories were only partly visible, and none named this work); no official repository was found. The only match for the title is a third-party course replication, which is excluded.

The paper states that the deep models follow "the original implementations" §3.1.2. Which implementation and version each model used is not stated.

Pitfalls

• Metric naming conflicts. Section 3.1.2 says regression is reported as RMSE, but the Table 2 caption and the Training paragraph say MSE. The caption also says cross-entropy values are multiplied by 100, so they are not raw cross-entropy §3.1.2 tab. 2.
• Abstract vs body. The abstract says XGBoost outperforms the deep models "across the datasets". Section 3.2 says 8 of 11 unseen datasets, and Table 2 shows TabNet ahead of XGBoost on Rossmann, CoverType and Higgs §3.2 tab. 2.
• Relative-performance figures. The text gives 3.4% (XGBoost), 11.8% (DNF-Net) and 7.5% (1D-CNN). Table 3 gives 3.34, 11.96 and 7.56. The text also calls XGBoost second best, but Table 3 places the Simple Ensemble (3.15%) ahead of it §3.2 tab. 3.
• Averaging set unclear. The caption says "unseen datasets", and the text says "all its unseen datasets (geometric mean)". It is not stated which datasets are averaged for XGBoost and the classical ensemble, which have no paper of origin §3.2 tab. 3.
• Ensemble weights. Eq. 2 uses the validation loss directly as the weight, not a normalized one, even though the text says "normalized". Read literally, models with higher loss get larger weights. The text says low-loss models are favoured when ordering subsets, so the intent is probably the opposite. The implementation is not given; choose and state one §2.2 eq. 2 §3.2.
• Eq. 1 index. The sum runs over k but the summand is written with θ_m. Read it as a sum over the K members §2.2 eq. 1.
• Ensemble members not fully defined. The classical ensemble is XGBoost, SVM and CatBoost §3.2. The membership of "Deep Ensemble w/o XGBoost" is not stated explicitly, and the combination rule (eq. 1 or eq. 2) used for either ensemble in Table 2 is not stated §3.2 tab. 2.
• Uncertainty measure undefined. Subset selection by per-example uncertainty names no measure ("some uncertainty measure") §3.2.
• Hyperparameter counts. The text says 6 to 9 main hyperparameters per model. The appendix lists 6 (CatBoost), 10 (XGBoost), 7 (NODE), 7 (TabNet), 4 (DNF-Net) and 4 (1D-CNN) §3.1.2 app. B.
• Duplicate NODE learning rate. NODE lists two learning-rate ranges, [e^-5, 1] and [e^-4, 0.5]. Which one applies is not stated app. B.
• Library names. Several XGBoost and CatBoost parameters have names such as "Eta", "Max size" and "Random strength". The paper does not map them to library arguments or state the library versions app. B.
• Hyperopt algorithm. The paper says Bayesian optimization with Hyperopt, but not which Hyperopt estimator was used. Results may depend on it, and the authors say so §3.1.2 §3.2.
• Split conflicts. The appendix puts "Year" in both the 80/20 list and the 70/10/20 list. Gas has no split rule. Whether the datasets other than Rossmann and Higgs are retrained on the full training set after tuning is not stated app. B.
• Repetition count. Section 3.1.2 says random splits use three partitions and others four seeds, while the Table 2 caption says four training runs. Whether random-split datasets get 3 × 4 runs or 3 runs is not stated §3.1.2 tab. 2.
• Significance threshold. Section 3.1.2 sets the level at 95% (p < 0.05). The results text quotes p < 0.005 §3.1.2 §3.2. The pairwise procedure after the omnibus Friedman test is not named §3.1.2.
• Categorical features. Section 3.1.1 says some datasets have categorical features. The encoding is not described, and the features and classes columns of tab. 1 do not separate categorical from numerical §3.1.1 tab. 1.
• Cost claims are not measured. The "more than an order of magnitude" runtime gap comes with no numbers. The iteration-count comparison is shown only for Shrutime §3.2 fig. 2.
• Hardware and software not stated. No GPU, CPU, library versions or training times are given, so the runtime comparison cannot be reproduced §3.1.2 §3.2.
• Limited sample. The authors note that eleven datasets cannot represent all tabular data §1. The 1D-CNN model was designed for a Kaggle competition dataset that the authors did not use, so its "unseen" status is by construction §2.1 §3.1.1.
• Author-reported. All numbers come from the authors' own runs, with no independent replication and no code to rerun.

Lineage

• Builds on: XGBoost §2, CatBoost and SVM for the classical ensemble §3.2, Hyperopt for tuning §3.1.2, Friedman's test for significance §3.1.2.
• Compared against, not carded here: TabNet, NODE, DNF-Net and 1D-CNN §2.1. The 1D-CNN is the Kaggle solution by Baosenguo, cited as a GitHub repository §2.1.
• Superseded by: not stated in this wiki.