Gradient-boosted decision trees

Overview

Gradient-boosted decision trees (GBDT) are the tree-based models the paper treats as the reference for tabular data, meaning one row per record and one column per feature 2106.03253 §1. The paper uses XGBoost as its implementation and baseline 2106.03253 §2 §3.2. The cited sources do not describe the boosting procedure itself, so this page covers only what the benchmarks measure and what they leave unstated.

A second paper, 1909.06312, is not boosting. It builds a neural network out of differentiable decision trees and compares it with CatBoost and XGBoost 1909.06312 §4.1. It is listed here because it is the main alternative to GBDT in the same tabular setting.

A third paper, 1802.00752, is a different setting: image classification of 400 histology slides. It uses LightGBM only as the final classifier over features from frozen image networks 1802.00752 §2.5. Its boosting values come from the code, not the paper, so they are listed separately below.

Variants

Three variants have cards in this wiki. Other libraries are named in the papers but are not compiled here.

XGBoost

The single-model baseline. Each dataset is tuned with Hyperopt for 1,000 steps over the search space in app. B 2106.03253 §3.1.2. On the test set it was usually the best single model, with a mean relative deterioration of 3.34% 2106.03253 tab. 3. Card: Tabular Data: Deep Learning is Not All You Need.

LightGBM on pooled CNN features

The classifier in a histology pipeline. Its trees see one feature vector per augmented view of a slide, not raw pixels. The features come from three frozen ImageNet encoders, and the paper trains one multiclass model per encoder, crop size, fold and seed 1802.00752 §2.4 §2.5. The authors say that fine-tuning a pretrained network worked poorly for them, but give no numbers 1802.00752 §2.2. The fused blend reaches 87.2% four-class accuracy under 10-fold cross-validation tab. 1. Card: Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis.

CatBoost in the classical ensemble

CatBoost is tuned with the same protocol 2106.03253 app. B. It appears only inside the "Simple Ensemble" with XGBoost and SVM, which scored 3.15% 2106.03253 §3.2 tab. 3. No separate CatBoost result is reported tab. 2.

Differentiable oblivious trees (NODE)

NODE stacks layers of oblivious trees, where every level of a tree uses the same feature and threshold, so a tree is a table of responses [arxiv:1909.06312 §2]. The feature choice and each split are soft (entmax with alpha = 1.5), and all parameters are fit together by mini-batch gradient descent [arxiv:1909.06312 §3.1] §3.3. The paper links the design to the oblivious trees used in CatBoost 1909.06312 §1. Implement entmax as the paper states. The official repository's class default is a different function, so check its card before reusing that code: Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data.

When to use which

The 2021 benchmark covers only the 11 datasets in tab. 1, with 7k to 1,018k samples and 10 to 2,000 features 2106.03253 tab. 1. Hardware, training time and library versions are not stated 2106.03253 §3.2. The 2019 paper covers six datasets and reports its own timings 1909.06312 tab. 4.

Hyperparameters

XGBoost

The XGBoost search space as written in the appendix. "Log-uniform [a, b]" is a log-scale uniform draw, and "Discrete uniform" draws integers [arxiv:2106.03253 app. B]. The paper does not map these names to library arguments, so check the library documentation before use 2106.03253 app. B.

Shared protocol 2106.03253 §3.1.2: 1,000 search steps per dataset, selection by smallest validation loss, and test results as the mean of four seeds with standard error of the mean. Inputs are standardized using training statistics only 2106.03253 §3.1.1.

LightGBM (histology pipeline)

The paper names LightGBM but gives almost no values 1802.00752 §2.5. The values below come from the authors' code at commit 3eedb3c, so they are not stated in the paper train_lgbm.py.

Bagging is disabled, so the bagging fraction has no effect. The code's log comments give 0.66 instead train_lgbm.py. Folds are 10 stratified splits at image level, so the 50 descriptors of one image stay together 1802.00752 §2.5. The paper's folds come from a committed file, and regenerating them does not reproduce the reported numbers [code: utils.py#L3, #L83].

NODE

The two papers search different spaces for NODE. Neither is a default.

The 2019 paper fixes the learning rate at 1e-3 1909.06312 app. A.2.3. The 2021 benchmark lists two learning-rate ranges for NODE [arxiv:2106.03253 app. B].

Results

2021 benchmark

Author-reported, on 11 datasets, lower is better 2106.03253 tab. 2 tab. 3.

• Mean relative deterioration: XGBoost 3.34%, the Simple Ensemble 3.15%, and the deep ensemble with XGBoost 2.32%. The deep ensemble without XGBoost scored 6.91% 2106.03253 tab. 3. The averages are taken over each model's unseen datasets.
• Unseen datasets: XGBoost beat the deep models on 8 of the 11 datasets that their own papers did not use, with p<0.005 2106.03253 §3.2.
• Per dataset: XGBoost was best on Eye Movements (56.07 ± 0.65). The deep ensemble with XGBoost was best on CoverType, Gesture, YearPrediction, MSLR, Shrutime and Blastchar 2106.03253 tab. 2.
• Search cost: on Shrutime, XGBoost reached good performance in fewer search iterations. The authors also report XGBoost training as more than an order of magnitude faster, without figures 2106.03253 §3.2 fig. 2.

2019 paper (NODE)

Author-reported, on six datasets, ten seeds per method. Lower is better 1909.06312 tab. 1 tab. 2.

• Default hyperparameters: NODE had the best value on all six datasets, against default CatBoost and XGBoost 1909.06312 tab. 1.
• Tuned hyperparameters: NODE was best on Epsilon, YearPrediction, Higgs and Click. Tuned XGBoost was best on Microsoft and Yahoo [arxiv:1909.06312 tab. 2]. Tuned XGBoost was therefore ahead on two of the six datasets.
• Runtime: the authors describe NODE inference as on par with heavily optimised GBDT libraries, while its training time is longer [arxiv:1909.06312 tab. 4].

2018 histology paper (LightGBM)

Author-reported, on the 400 ICIAR 2018 images, under 10-fold stratified cross-validation 1802.00752 §2.1 §3.

• Four-class, blended: 87.2 ± 2.6% accuracy, from the average of six encoder and crop models. The best single model, ResNet at 400 px, scored 84.2 ± 4.2% 1802.00752 tab. 1.
• Two-class (non-carcinoma vs carcinoma): 93.8 ± 2.3% accuracy and ROC AUC 0.973 1802.00752 §3 fig. 4a. The paper does not say how this task is derived. The code merges the four-class blend into two classes crossvalidate_blending.py, so these numbers come from that merge, not from a separate binary model.
• The paper gives no numeric comparison with earlier methods 1802.00752 §4.

Implementation sketch

The sketch covers the outer tuning loop used in the 2021 benchmark. NODE's inner training is different and is covered in its card 1909.06312 §3.3.

# Tune XGBoost on one dataset  [arxiv:2106.03253 §3.1.2] [app. B]
# Input:  train and valid splits
# Output: configuration with the smallest validation loss

def tune_xgboost(train, valid):
    best_config, best_loss = None, math.inf
    history = []
    # 1,000 search steps per dataset  [§3.1.2]
    for step in range(1000):
        # Propose from the search space in [app. B]; estimator not stated
        config = propose(XGBOOST_SPACE, history)
        loss = validation_loss(fit_xgboost(config, train), valid)
        history.append((config, loss))
        # Keep the smallest validation loss  [app. B]
        if loss < best_loss:
            best_config, best_loss = config, loss
    return best_config

Contradictions

• Scope of the win. The abstract says XGBoost outperforms the deep models "across the datasets". Section 3.2 says 8 of 11 unseen datasets, and Table 2 shows TabNet ahead of XGBoost on Rossmann, CoverType and Higgs 2106.03253 §3.2 tab. 2.
• Relative figures. The text gives 3.4%, 11.8% and 7.5% for XGBoost, DNF-Net and 1D-CNN. Table 3 gives 3.34, 11.96 and 7.56 2106.03253 §3.2 tab. 3.
• Rank of XGBoost. The text calls XGBoost second best. Table 3 places the Simple Ensemble (3.15) ahead of XGBoost (3.34) 2106.03253 §3.2 tab. 3.
• Significance level. Section 3.1.2 sets the Friedman test at 95% (p < 0.05). Section 3.2 quotes p < 0.005 2106.03253 §3.1.2 §3.2.
• NODE against tuned XGBoost. The 2019 paper reports NODE ahead on four of six datasets 1909.06312 tab. 2. The 2021 benchmark reports NODE at 14.21% mean relative deterioration, the worst of its four deep models, against 3.34% for XGBoost 2106.03253 tab. 3. Both papers put NODE ahead of XGBoost on YearPrediction 1909.06312 tab. 2 2106.03253 tab. 2. The averages cover different datasets and tuning budgets. The 2019 paper uses 50 TPE steps for CatBoost and XGBoost 1909.06312 app. A.2, and the 2021 benchmark uses 1,000 Hyperopt steps 2106.03253 §3.1.2.
• Choice function ablation. The 2019 paper's text says entmax was best in all experiments 1909.06312 §4.2. Its own table shows sparsemax ahead of entmax on YearPrediction with two layers, 76.81 against 77.05 1909.06312 tab. 3.
• Epsilon metric. The 2019 paper labels Epsilon as classification error 1909.06312 §4.1. The 2021 benchmark labels it cross-entropy × 100 2106.03253 tab. 2. Check the label before comparing the two numbers.
• Fusion gain (2018 paper). The text says the fused model beats every constituent by 4 to 5 points. Table 1 gives a gap of 3.0 to 4.7 points against the individual model means 1802.00752 §3 tab. 1. The text also gives 600 models, while the code trains 300 1802.00752 §2.5 train_lgbm.py.

Papers

• Tabular Data: Deep Learning is Not All You Need: compares four deep models with XGBoost on 11 tabular datasets under one tuning protocol, and reports that an ensemble of both is best on the author-reported mean.
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data: proposes NODE, a differentiable stack of oblivious trees, and reports it ahead of tuned XGBoost and CatBoost on four of six datasets.
• Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis: uses LightGBM as the final classifier over pooled features from frozen ImageNet encoders, and reports 87.2% four-class accuracy under 10-fold cross-validation (author-reported).