--- kind: concept slug: gradient-boosted-decision-trees title: Gradient-boosted decision trees summary: >- Tree ensembles trained by gradient boosting. XGBoost was usually the best single model and the quickest to tune in a 2021 benchmark on 11 tabular datasets. LightGBM served as the final classifier in a 2018 histology study. NODE, a 2019 differentiable tree model, reports beating tuned XGBoost on four of six datasets. tags: [tabular-data, ensembles, benchmark, decision-trees, transfer-learning] papers: ["1802.00752", "1909.06312", "2106.03253"] related: [ensemble-learning, hyperparameter-optimization, bayesian-optimization, transfer-learning] status: draft updated_at: 2026-11-01 --- ## Overview Gradient-boosted decision trees (GBDT) are the tree-based models the paper treats as the reference for tabular data, meaning one row per record and one column per feature [arxiv:2106.03253 §1]. The paper uses XGBoost as its implementation and baseline [arxiv:2106.03253 §2] [§3.2]. The cited sources do not describe the boosting procedure itself, so this page covers only what the benchmarks measure and what they leave unstated. A second paper, [arxiv:1909.06312], is not boosting. It builds a neural network out of differentiable decision trees and compares it with CatBoost and XGBoost [arxiv:1909.06312 §4.1]. It is listed here because it is the main alternative to GBDT in the same tabular setting. A third paper, [arxiv:1802.00752], is a different setting: image classification of 400 histology slides. It uses LightGBM only as the final classifier over features from frozen image networks [arxiv:1802.00752 §2.5]. Its boosting values come from the code, not the paper, so they are listed separately below. ## Variants Three variants have cards in this wiki. Other libraries are named in the papers but are not compiled here. ### XGBoost The single-model baseline. Each dataset is tuned with Hyperopt for 1,000 steps over the search space in [app. B] [arxiv:2106.03253 §3.1.2]. On the test set it was usually the best single model, with a mean relative deterioration of 3.34% [arxiv:2106.03253 tab. 3]. Card: [Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253). ### LightGBM on pooled CNN features The classifier in a histology pipeline. Its trees see one feature vector per augmented view of a slide, not raw pixels. The features come from three frozen ImageNet encoders, and the paper trains one multiclass model per encoder, crop size, fold and seed [arxiv:1802.00752 §2.4] [§2.5]. The authors say that fine-tuning a pretrained network worked poorly for them, but give no numbers [arxiv:1802.00752 §2.2]. The fused blend reaches 87.2% four-class accuracy under 10-fold cross-validation [tab. 1]. Card: [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752). ### CatBoost in the classical ensemble CatBoost is tuned with the same protocol [arxiv:2106.03253 app. B]. It appears only inside the "Simple Ensemble" with XGBoost and SVM, which scored 3.15% [arxiv:2106.03253 §3.2] [tab. 3]. No separate CatBoost result is reported [tab. 2]. ### Differentiable oblivious trees (NODE) NODE stacks layers of oblivious trees, where every level of a tree uses the same feature and threshold, so a tree is a table of responses [arxiv:1909.06312 §2]. The feature choice and each split are soft (entmax with alpha = 1.5), and all parameters are fit together by mini-batch gradient descent [arxiv:1909.06312 §3.1] [§3.3]. The paper links the design to the oblivious trees used in CatBoost [arxiv:1909.06312 §1]. Implement entmax as the paper states. The official repository's class default is a different function, so check its card before reusing that code: [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](/papers/1909.06312). ## When to use which The 2021 benchmark covers only the 11 datasets in [tab. 1], with 7k to 1,018k samples and 10 to 2,000 features [arxiv:2106.03253 tab. 1]. Hardware, training time and library versions are not stated [arxiv:2106.03253 §3.2]. The 2019 paper covers six datasets and reports its own timings [arxiv:1909.06312 tab. 4]. | Situation | Use | Why | |---|---|---| | Tabular data, one model, limited tuning time | XGBoost | Usually the best single model [arxiv:2106.03253 §3.2]. On Shrutime it reached good performance in fewer search iterations than the deep models [arxiv:2106.03253 fig. 2] | | Tabular data, you can train several models, and accuracy on average matters most | Ensemble of XGBoost with the deep models | Lowest author-reported mean, 2.32% against 3.34% for XGBoost [arxiv:2106.03253 tab. 3]. Compute cost of the ensemble is not stated | | Training and inference time matter | XGBoost | On YearPrediction, the 2019 paper reports NODE training in 7 min 42 s on a 1080Ti and XGBoost in 1 min 13 s on the same GPU. Inference was 8.56 s for NODE and 4.45 s for XGBoost [arxiv:1909.06312 tab. 4] | | You need a differentiable tree model that can train inside a larger network | NODE | This is the use case the 2019 paper names for it [arxiv:1909.06312 §5] | | Learning-to-rank data such as Yahoo or Microsoft | Tuned XGBoost | In the 2019 paper's tuned regime NODE did not win there. The authors suspect oblivious trees are a poor inductive bias for Yahoo [arxiv:1909.06312 §4.1] [tab. 2] | | You need the best result on one specific dataset | Check that dataset's row in [tab. 2] | The ensemble was not best everywhere. XGBoost alone was best on Eye Movements [arxiv:2106.03253 tab. 2] | | Few labelled images (hundreds), a strong pretrained encoder, and many augmented views affordable | LightGBM on pooled features from frozen encoders | This is the only setting the 2018 paper reports, with 400 images [arxiv:1802.00752 §2.1]. It gives no numeric comparison with other classifiers on the same features [arxiv:1802.00752 §3], so the tree model's own contribution is not measured | | Deep network as the only model | Depends on the paper's protocol | The 2021 benchmark reports single deep models at 7.56% to 14.21% mean relative deterioration [arxiv:2106.03253 tab. 3]. The 2019 paper reports NODE ahead of tuned XGBoost on four of six datasets [arxiv:1909.06312 tab. 2]. See Contradictions | ## Hyperparameters ### XGBoost The XGBoost search space as written in the appendix. "Log-uniform [a, b]" is a log-scale uniform draw, and "Discrete uniform" draws integers [arxiv:2106.03253 app. B]. The paper does not map these names to library arguments, so check the library documentation before use [arxiv:2106.03253 app. B]. | Hyperparameter | Search space | |---|---| | Number of estimators | Uniform [100, 4000] | | Eta | Log-uniform [e^-7, 1] | | Max depth | Discrete uniform [1, 10] | | Subsample | Uniform [0.2, 1] | | Colsample bytree | Uniform [0.2, 1] | | Colsample bylevel | Uniform [0.2, 1] | | Min child weight | Log-uniform [e^-16, e^5] | | Alpha | Uniform choice of {0, log-uniform [e^-16, e^2]} | | Lambda | Uniform choice of {0, log-uniform [e^-16, e^2]} | | Gamma | Uniform choice of {0, log-uniform [e^-16, e^2]} | Shared protocol [arxiv:2106.03253 §3.1.2]: 1,000 search steps per dataset, selection by smallest validation loss, and test results as the mean of four seeds with standard error of the mean. Inputs are standardized using training statistics only [arxiv:2106.03253 §3.1.1]. ### LightGBM (histology pipeline) The paper names LightGBM but gives almost no values [arxiv:1802.00752 §2.5]. The values below come from the authors' code at commit `3eedb3c`, so they are not stated in the paper [code: train_lgbm.py#L61-L73]. | Hyperparameter | Value in code | Where | |---|---|---| | Objective | Multiclass, 4 classes | [code: train_lgbm.py#L64-L65] | | Learning rate | 0.1 | [code: train_lgbm.py#L61] | | Boosting rounds | 70, fixed, no early stopping | [code: train_lgbm.py#L62, #L93-L95] | | num_leaves | 191 | [code: train_lgbm.py#L69] | | max_depth | 7 | [code: train_lgbm.py#L73] | | feature_fraction | 0.46 | [code: train_lgbm.py#L70] | | bagging_fraction | 0.69, no effect (see below) | [code: train_lgbm.py#L71] | | bagging_freq | 0, bagging off | [code: train_lgbm.py#L72] | | Seeds per fold | 5 | [arxiv:1802.00752 §2.5] [code: train_lgbm.py#L21] | Bagging is disabled, so the bagging fraction has no effect. The code's log comments give 0.66 instead [code: train_lgbm.py#L138]. Folds are 10 stratified splits at image level, so the 50 descriptors of one image stay together [arxiv:1802.00752 §2.5]. The paper's folds come from a committed file, and regenerating them does not reproduce the reported numbers [code: utils.py#L3, #L83]. ### NODE The two papers search different spaces for NODE. Neither is a default. | Hyperparameter | 2019 paper, tuned grid [arxiv:1909.06312 app. A.2.3] | 2021 benchmark search space [arxiv:2106.03253 app. B] | |---|---|---| | Layers | {2, 4, 8} | Discrete uniform [1, 10] | | Total trees | {1024, 2048}, counted across all layers | Choice of {256, 512, 1024, 2048} | | Tree depth | {6, 8} | Discrete uniform [4, 9] | | Tree output dimension | {2, 3} | Discrete uniform [1, 5] | | Batch size | Largest that fits GPU memory | Uniform choice of {512, 1024, 2048, 4096, 8192} | The 2019 paper fixes the learning rate at 1e-3 [arxiv:1909.06312 app. A.2.3]. The 2021 benchmark lists two learning-rate ranges for NODE [arxiv:2106.03253 app. B]. ## Results ### 2021 benchmark Author-reported, on 11 datasets, lower is better [arxiv:2106.03253 tab. 2] [tab. 3]. - **Mean relative deterioration:** XGBoost 3.34%, the Simple Ensemble 3.15%, and the deep ensemble with XGBoost 2.32%. The deep ensemble without XGBoost scored 6.91% [arxiv:2106.03253 tab. 3]. The averages are taken over each model's unseen datasets. - **Unseen datasets:** XGBoost beat the deep models on 8 of the 11 datasets that their own papers did not use, with p<0.005 [arxiv:2106.03253 §3.2]. - **Per dataset:** XGBoost was best on Eye Movements (56.07 ± 0.65). The deep ensemble with XGBoost was best on CoverType, Gesture, YearPrediction, MSLR, Shrutime and Blastchar [arxiv:2106.03253 tab. 2]. - **Search cost:** on Shrutime, XGBoost reached good performance in fewer search iterations. The authors also report XGBoost training as more than an order of magnitude faster, without figures [arxiv:2106.03253 §3.2] [fig. 2]. ### 2019 paper (NODE) Author-reported, on six datasets, ten seeds per method. Lower is better [arxiv:1909.06312 tab. 1] [tab. 2]. - **Default hyperparameters:** NODE had the best value on all six datasets, against default CatBoost and XGBoost [arxiv:1909.06312 tab. 1]. - **Tuned hyperparameters:** NODE was best on Epsilon, YearPrediction, Higgs and Click. Tuned XGBoost was best on Microsoft and Yahoo [arxiv:1909.06312 tab. 2]. Tuned XGBoost was therefore ahead on two of the six datasets. - **Runtime:** the authors describe NODE inference as on par with heavily optimised GBDT libraries, while its training time is longer [arxiv:1909.06312 tab. 4]. ### 2018 histology paper (LightGBM) Author-reported, on the 400 ICIAR 2018 images, under 10-fold stratified cross-validation [arxiv:1802.00752 §2.1] [§3]. - **Four-class, blended:** 87.2 ± 2.6% accuracy, from the average of six encoder and crop models. The best single model, ResNet at 400 px, scored 84.2 ± 4.2% [arxiv:1802.00752 tab. 1]. - **Two-class (non-carcinoma vs carcinoma):** 93.8 ± 2.3% accuracy and ROC AUC 0.973 [arxiv:1802.00752 §3] [fig. 4a]. The paper does not say how this task is derived. The code merges the four-class blend into two classes [code: crossvalidate_blending.py#L154-L162], so these numbers come from that merge, not from a separate binary model. - The paper gives no numeric comparison with earlier methods [arxiv:1802.00752 §4]. ## Implementation sketch The sketch covers the outer tuning loop used in the 2021 benchmark. NODE's inner training is different and is covered in its card [arxiv:1909.06312 §3.3]. ```pseudocode # Tune XGBoost on one dataset [arxiv:2106.03253 §3.1.2] [app. B] # Input: train and valid splits # Output: configuration with the smallest validation loss def tune_xgboost(train, valid): best_config, best_loss = None, math.inf history = [] # 1,000 search steps per dataset [§3.1.2] for step in range(1000): # Propose from the search space in [app. B]; estimator not stated config = propose(XGBOOST_SPACE, history) loss = validation_loss(fit_xgboost(config, train), valid) history.append((config, loss)) # Keep the smallest validation loss [app. B] if loss < best_loss: best_config, best_loss = config, loss return best_config ``` ## Contradictions - **Scope of the win.** The abstract says XGBoost outperforms the deep models "across the datasets". Section 3.2 says 8 of 11 unseen datasets, and Table 2 shows TabNet ahead of XGBoost on Rossmann, CoverType and Higgs [arxiv:2106.03253 §3.2] [tab. 2]. - **Relative figures.** The text gives 3.4%, 11.8% and 7.5% for XGBoost, DNF-Net and 1D-CNN. Table 3 gives 3.34, 11.96 and 7.56 [arxiv:2106.03253 §3.2] [tab. 3]. - **Rank of XGBoost.** The text calls XGBoost second best. Table 3 places the Simple Ensemble (3.15) ahead of XGBoost (3.34) [arxiv:2106.03253 §3.2] [tab. 3]. - **Significance level.** Section 3.1.2 sets the Friedman test at 95% (p < 0.05). Section 3.2 quotes p < 0.005 [arxiv:2106.03253 §3.1.2] [§3.2]. - **NODE against tuned XGBoost.** The 2019 paper reports NODE ahead on four of six datasets [arxiv:1909.06312 tab. 2]. The 2021 benchmark reports NODE at 14.21% mean relative deterioration, the worst of its four deep models, against 3.34% for XGBoost [arxiv:2106.03253 tab. 3]. Both papers put NODE ahead of XGBoost on YearPrediction [arxiv:1909.06312 tab. 2] [arxiv:2106.03253 tab. 2]. The averages cover different datasets and tuning budgets. The 2019 paper uses 50 TPE steps for CatBoost and XGBoost [arxiv:1909.06312 app. A.2], and the 2021 benchmark uses 1,000 Hyperopt steps [arxiv:2106.03253 §3.1.2]. - **Choice function ablation.** The 2019 paper's text says entmax was best in all experiments [arxiv:1909.06312 §4.2]. Its own table shows sparsemax ahead of entmax on YearPrediction with two layers, 76.81 against 77.05 [arxiv:1909.06312 tab. 3]. - **Epsilon metric.** The 2019 paper labels Epsilon as classification error [arxiv:1909.06312 §4.1]. The 2021 benchmark labels it cross-entropy × 100 [arxiv:2106.03253 tab. 2]. Check the label before comparing the two numbers. - **Fusion gain (2018 paper).** The text says the fused model beats every constituent by 4 to 5 points. Table 1 gives a gap of 3.0 to 4.7 points against the individual model means [arxiv:1802.00752 §3] [tab. 1]. The text also gives 600 models, while the code trains 300 [arxiv:1802.00752 §2.5] [code: train_lgbm.py#L15, #L21]. ## Papers - [Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253): compares four deep models with XGBoost on 11 tabular datasets under one tuning protocol, and reports that an ensemble of both is best on the author-reported mean. - [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](/papers/1909.06312): proposes NODE, a differentiable stack of oblivious trees, and reports it ahead of tuned XGBoost and CatBoost on four of six datasets. - [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752): uses LightGBM as the final classifier over pooled features from frozen ImageNet encoders, and reports 87.2% four-class accuracy under 10-fold cross-validation (author-reported).