--- kind: concept slug: ensemble-learning title: Ensemble learning summary: >- Combine several models into one predictor. Bagging and boosting over trees, stacking with a meta-model, and fusing the outputs of deep or LLM members are the recurring designs. Author-reported gains are often small or from one dataset. In an 11-dataset tabular benchmark, boosted trees were usually the strongest single model, and a mix with deep networks scored best. tags: [ensembles, bagging, boosting, stacking, tabular-data, llm-ensembles, federated-learning, medical-imaging] papers: ["1305.1707", "1802.00752", "1909.06312", "2006.07242", "2104.07551", "2105.00108", "2106.03253", "2110.03224", "2206.06602", "2212.12343", "2205.14090", "2304.02858", "2304.13029", "2306.02561", "2506.08274", "2608.07471", "2609.10524", "2605.25305"] related: [ensembles, stacked-generalization, random-forest, gradient-boosted-decision-trees, adaboost, ensemble-clustering, knowledge-distillation, federated-aggregation, feature-scaling, isolation-forest, bayesian-optimization, llm-as-a-judge, time-series-classification, time-series-forecasting, random-convolutional-kernels, transfer-learning] status: draft updated_at: 2026-11-01 --- ## Overview An ensemble averages, votes or fuses several models so that the combined predictor beats its members. The cards cover four basic moves: train members on different samples (bagging), train each member to fix the previous one's errors (boosting), learn how to combine member outputs (stacking), or pick a member per input (dynamic selection, LLM ranking). Most cards are empirical. Each one reports gains for its own datasets and tuning budget, so the page records which conditions each gain was measured under. ## Variants ### Bagging and random forests Members are trained on resampled rows and their outputs are averaged. Random forests are the usual tree form [[random-forest]]. Bagging also supplies the candidate pool for dynamic selection: one benchmark builds 100 bagged perceptrons and selects among them per query [arxiv:2212.12343 §3.2] [tab. 1]. Card: [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108) notes that a bagged or boosted ensemble is linear in its members, so it can be explained directly [arxiv:2105.00108 app. A.5.4]. ### Boosting Trees are added one at a time, each fitted to the errors of the ones before. The most used implementation in the cards is XGBoost, which the tabular benchmark found usually the best single model and the quickest to tune [arxiv:2106.03253 §3.2] [[gradient-boosted-decision-trees]]. Card: [Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253). For class imbalance, a review lists RUSBoost, which combines random under-sampling with boosting, and says it is simpler and faster than SMOTEBoost. The review runs no experiments to check this [arxiv:1305.1707 §3] [[adaboost]]. Card: [Class Imbalance Problem in Data Mining Review](/papers/1305.1707). Differentiable variants replace the hard trees with soft ones trained by gradient descent. NODE is one; it beats default CatBoost and XGBoost on all six of its datasets, and tuned versions on four of six, but trains and infers more slowly than the GBDT baselines [arxiv:1909.06312 §4.1] [tab. 4]. Card: [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](/papers/1909.06312). ### Stacking and voting A meta-model learns from the base models' outputs; voting uses a fixed rule. Three cards test these on imbalanced, fraud and image-feature data, with different sets and metrics. A review and benchmark shows Stacking-II above Stacking-I on its AUC heatmaps, and its best named pairs are gradient-boosting models with resampling [arxiv:2304.02858 §5.3] [arxiv:2304.02858 §7]. Card: [Imbalance benchmark pairing augmentation with ensembles](/papers/2304.02858). A rice-variety study stacks four fine-tuned CNNs under an XGBoost meta-model and reports 1.000 accuracy on its own set [arxiv:2609.10524 §4.4] [tab. 6]. Card: [Precision in Rice Variety Classification using Stacking-Based Ensemble Learning](/papers/2609.10524). On card-fraud data, a stack reaches AUC 0.954 against 0.946 for the best single model, logistic regression [arxiv:2608.07471 §5]. Card: [Application of Artificial Intelligence for Fraudulent Banking Operations Recognition](/papers/2608.07471). See [[stacked-generalization]] for the stacking details. ### Ensembles of deep or mixed models on tabular data Deep networks can be ensembled with each other or with a tree model. On 11 datasets, a validation-loss-weighted ensemble of four deep models plus XGBoost scored 2.32% mean relative deterioration (lower is better). XGBoost alone scored 3.34%, and the deep models without XGBoost 6.91% [arxiv:2106.03253 §3.2] [tab. 3]. The weighting formula has a sign ambiguity, so reproduce it with care [arxiv:2106.03253 §2.2] [eq. 2]. ### Ensemble distillation in federated learning FedDF replaces FedAvg's weight averaging with distillation. The server trains one model to match the averaged logits of the client models on unlabelled data [arxiv:2006.07242 §3] [eq. 1]. It needed about 20 rounds on CIFAR-10 against 104 for FedAvg, with α = 1, 40 local epochs and 40% participation [arxiv:2006.07242 tab. 1]. The server still uses labelled validation data for early stopping [arxiv:2006.07242 §4.1]. Card: [Ensemble Distillation for Robust Model Fusion in Federated Learning](/papers/2006.07242). See [[knowledge-distillation]] and [[federated-aggregation]]. ### Averaging over encoders, crops and folds Frozen ImageNet encoders give fixed features; one LightGBM model is fitted per encoder, crop size, cross-validation fold and seed, and the predicted probabilities are averaged over augmentations, seeds and the six encoder/crop models [arxiv:1802.00752 §2.5]. On 400 histology images with 10-fold cross-validation, the author-reported four-class accuracy of the blend is 87.2 ± 2.6%. The six single models range from 82.5% to 84.2% mean accuracy, and the best of them, ResNet at 400 px, reaches 84.2 ± 4.2% [arxiv:1802.00752 tab. 1]. The two-class task is reported at 93.8% without a baseline [arxiv:1802.00752 §3]. Card: [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752). See [[transfer-learning]] and [[gradient-boosted-decision-trees]]. ### One network that acts as an ensemble A batched ensemble of random, untrained representations can run as one network. Deep Isolation Forest stacks 50 representations with six isolation trees each, scoring points by path depth [arxiv:2206.06602 §5.2.1] [eq. 3-5]. Its author-reported AUC-ROC of 0.925 on ten tabular sets is above the deep ensembles' 0.889 [arxiv:2206.06602 tab. 4]. Card: [Deep Isolation Forest for Anomaly Detection](/papers/2206.06602). See [[isolation-forest]]. ### Mixtures of models for Bayesian optimization EGP-TS keeps one Gaussian process per kernel in a dictionary and weights them by fit. Each query is chosen by Thompson sampling from the weighted mixture, using 50 random features per kernel [arxiv:2205.14090 §3] [arxiv:2205.14090 §3.1]. The author-reported regret is lower than single-GP Thompson sampling and two bandit ensembles [fig. 1–3]. Card: [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](/papers/2205.14090). See [[bayesian-optimization]]. ### Fusing LLM outputs LLM-Blender ranks N candidate answers pairwise, then fuses the top three into one answer. On MixInstruct it reaches GPT-Rank 3.01 against 3.90 for the best single model [arxiv:2306.02561 §2.3] [tab. 2]. The judge model is a ChatGPT version the paper does not name, so the rank is not reproducible from the paper alone. The released code does more pairwise passes than the paper states; plan the budget with the code's count [arxiv:2306.02561 §3.3]. Card: [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](/papers/2306.02561). See [[llm-as-a-judge]]. ### Hybrid classifiers for time series HIVE-COTE 2.0 (HC2) trains four modules on the same training set: the Shapelet Transform Classifier (STC), the Temporal Dictionary Ensemble (TDE), the Diverse Representation Canonical Interval Forest (DrCIF) and Arsenal, an ensemble of ROCKET-style classifiers [arxiv:2104.07551 §3] [§3.1] [§3.2] [§3.3] [§3.4]. Each module estimates its own accuracy from the training data. A new case gets a weighted sum of module probabilities, with each accuracy estimate raised to the power 4 [arxiv:2104.07551 §3]. The paper gives no formula for the combination. Card: [HIVE-COTE 2.0: a new meta ensemble for time series classification](/papers/2104.07551). Author-reported results, where a negative mean difference means HC2 is better. On 112 univariate UCR problems the mean difference is −1.06% against HC1, −1.36% against TS-CHIEF, −1.69% against InceptionTime and −2.49% against ROCKET [arxiv:2104.07551 tab. 3]. On 26 multivariate UEA problems the largest gap is −8.22% against DTW-D [arxiv:2104.07551 tab. 5]. The cost is run time: one sequential UCR resample takes about 340 hours for HC2, against about 2.85 hours for ROCKET [arxiv:2104.07551 tab. 4]. On the five problems where a sequential build takes over 12 hours, a 4-hour time contract reaches 98% of the full-build accuracy and a 12-hour contract 99% [arxiv:2104.07551 §7] [tab. 8]. An ablation over 11 variants built from the four modules finds the full ensemble significantly better than each of them [arxiv:2104.07551 §6] [tab. 7]. The Bake off redux re-run also ranks HC2 first on accuracy over 112 UCR problems, at 0.891, with MR-Hydra second at 0.884 [arxiv:2304.13029 §5] [tab. 14]. Its HC2 timing covers a different set of datasets, 263.89 hours in total over 142 problems [arxiv:2304.13029 tab. 19], so it is not a direct check on the 340-hour figure above. The authors suggest HC2 and MR-Hydra as defaults without prior knowledge [arxiv:2304.13029 §7]. Card: [Bake off redux](/papers/2304.13029). See [[time-series-classification]]. ### Ensemble models in a forecasting library Darts puts its ensemble models behind the same `fit` and `predict` calls as its single forecasters, and one of them learns how to combine its members with a regression model [arxiv:2110.03224 §2.6]. The paper reports no accuracy benchmark for any model, so it gives no evidence on how these ensembles compare with their members [arxiv:2110.03224 §3]. Card: [Darts: User-Friendly Modern Machine Learning for Time Series](/papers/2110.03224). See [[time-series-forecasting]]. ### Pushing a strong forecaster away from weaker ones The Weaker Separator Booster takes a tuned LSTM as the strong member and the mean of three weaker members (random forest, SVR and XGBoost) as a reference. It adds to the strong forecast a term proportional to the gap between the two, scaled by a weight that is zero at the first step of a 12-month horizon and peaks in the middle [arxiv:2605.25305 §3.2.3] [eq. 1-3]. On recursive 12-month tests of two campus electricity series, the author-reported MAE is 1990.87 kWh against 2264.50 for the LSTM alone at Palmas, and 464.93 against 466.92 at Coronel Vivida [tab. 14] [tab. 15]. The Coronel gain is under 0.5%, and each campus was tested on one year only. The global weight 0.0833 came from a random search over 0 to 1 [arxiv:2605.25305 §3.2.3]. Card: [Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations](/papers/2605.25305). See [[time-series-forecasting]]. ## When to use which | Situation | Use | Why | |---|---|---| | Tabular rows, one tree model allowed, little tuning time | Boosted trees (XGBoost) | Usually the strongest single model, and fewer search iterations [arxiv:2106.03253 §3.2] | | Tabular, budget for several models | Trees plus a deep model, averaged | Lowest mean error in the one benchmark that tested it [arxiv:2106.03253 tab. 3] | | Imbalanced classes, data can be resampled | Resampling plus boosting or a forest | Each pair ran 30 times; the best pair depends on the dataset [arxiv:2304.02858 §5.1] [arxiv:2304.02858 §5.3] | | Base models differ and a meta-model is affordable | Stacking with a meta-model | Meta-model learns how to combine base outputs [arxiv:2609.10524 §4.4] | | Scaling is unknown or costly | Tree ensembles | Largely insensitive to the scaler [arxiv:2506.08274 §5.2] [arxiv:2212.12343 §5.1] | | Several clients, weights cannot be averaged | Distillation of client models | Works across architectures; needs unlabelled data [arxiv:2006.07242 §3] | | Frozen features, few labelled images, and several encoders or crops available | Average LightGBM models over encoders, crops, folds and seeds | Fusion beat the best single model by 3.0 points on one 400-image set; the text claims 4 to 5 [arxiv:1802.00752 tab. 1] [§3] | | Forecasting, and a library with one API is wanted | Library ensemble model (e.g. Darts) | Convenience is the paper's claim; no accuracy evidence is given [arxiv:2110.03224 §2.6] [§3] | | Need one model at inference time | Single-network ensemble, or distillation | One network to deploy; the batched form tiles each mini-batch, so memory grows [arxiv:2206.06602 §5.2.1] | | Time series classification, accuracy first, and CPU hours are available | HC2, with a time contract if the budget is short | Most accurate in author-reported runs on 112 UCR and 26 UEA problems. A 4-hour contract keeps 98% of full-build accuracy on the five longest problems, but a full build is about 340 hours per resample [arxiv:2104.07551 tab. 3] [tab. 5] [§7] | | Several LLMs, one answer per input | Pairwise ranking, then fusion | Gains reported on one benchmark with a judge model [arxiv:2306.02561 tab. 2] | | Forecasting one series with a strong model and cheap weaker ones | Booster that moves the strong forecast away from the weaker mean | Author-reported gain over the strong model is small at one campus and untested beyond one year [arxiv:2605.25305 tab. 14] [tab. 15] | ## Implementation sketch Common core: fit members, combine their outputs. Stacking adds a second fit. The cards disagree on how the meta-model's training inputs are made. ```pseudocode # Fit each base model on the training split [arxiv:2304.02858 §5.1] for m in base_models: m.fit(X_train, y_train) # Soft voting: average the class probabilities [arxiv:2304.02858 §4.1] def soft_vote(models, x): return mean(m.predict_proba(x) for m in models) # Stacking: the meta-model learns from base outputs [arxiv:2609.10524 §4.4] # Open choice: out-of-fold or in-sample base outputs. Paper does not say. Z_train = stack_outputs(base_models, X_train) # shape: rows x (models*classes) meta.fit(Z_train, y_train) # Predict: base outputs, then meta-model [arxiv:2609.10524 §4.2] def stack_predict(x): return meta.predict(stack_outputs(base_models, x)) ``` ## Contradictions - **Deep networks vs trees.** The NODE paper reports wins over tuned XGBoost on four of six datasets [arxiv:1909.06312 §4.1]. The 11-dataset benchmark reports XGBoost ahead on 8 of 11 datasets outside the deep models' own papers [arxiv:2106.03253 §3.2]. Different datasets and tuning protocols. Neither is resolved here. - **Voting rule.** The benchmark text says hard voting is generally better than soft voting [arxiv:2304.02858 §5.3]. Its released code builds only soft voting models; see the Paper vs code section of its card. - **Stacking gains.** A 1.000 accuracy on a self-curated rice set [arxiv:2609.10524 tab. 6] and a 0.008 AUC gain on card-fraud data [arxiv:2608.07471 §5] are not comparable: different tasks, metrics and baselines. - **Fusion gain in one paper.** The histology paper says the fused model is 4 to 5 points better than any constituent [arxiv:1802.00752 §3]. Its Table 1 gives 87.2% against 84.2% for the best single model, a gap of 3.0 points [arxiv:1802.00752 tab. 1]. The two statements are not reconciled here. - **Within one paper.** The deep-model benchmark places the simple ensemble ahead of XGBoost in its table, but the text calls XGBoost second best [arxiv:2106.03253 tab. 3] [§3.2]. - **Booster result in one paper.** The Coronel Vivida MAE of the booster is 464.93 kWh in Table 15 and 465.02 kWh in the abstract [arxiv:2605.25305 tab. 15] [arxiv:2605.25305 abstract]. The difference is small, but the two values should be checked against each other before quoting either. - **Large-dataset runs in HC2.** The text says which HC2 components finished on FruitFlies and InsectSound, and the result tables report accuracies for different components [arxiv:2104.07551 §7] [tab. 10] [tab. 11]. Use those numbers only after checking the tables against the text. ## Papers - [Imbalance benchmark pairing augmentation with ensembles](/papers/2304.02858): stacking, voting and boosting with resampling, benchmarked on 27 sets. - [Class Imbalance Problem in Data Mining Review](/papers/1305.1707): survey of imbalance fixes, including boosting variants; no experiments. - [Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis](/papers/1802.00752): LightGBM over frozen CNN features, averaged over encoders, crops, folds and seeds. - [Darts: User-Friendly Modern Machine Learning for Time Series](/papers/2110.03224): forecasting library with ensemble models; no accuracy benchmark. - [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](/papers/1909.06312): differentiable oblivious-tree ensemble for tabular data. - [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108): attributions through stacked and ensembled models. - [Tabular Data: Deep Learning is Not All You Need](/papers/2106.03253): deep-model and XGBoost ensembles on 11 tabular datasets. - [Deep Isolation Forest for Anomaly Detection](/papers/2206.06602): batched ensemble of random representations, one network. - [The choice of scaling technique matters for classification performance](/papers/2212.12343): scaler sensitivity of bagging, dynamic selection and other ensembles. - [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](/papers/2205.14090): ensemble of Gaussian processes as a surrogate. - [Bake off redux](/papers/2304.13029): hybrid ensemble HC2 among time-series classifiers. - [HIVE-COTE 2.0: a new meta ensemble for time series classification](/papers/2104.07551): four-module meta ensemble (STC, TDE, DrCIF, Arsenal) on UCR and UEA problems. - [LLM-Blender](/papers/2306.02561): pairwise ranking and fusion of LLM answers. - [Ensemble Distillation for Robust Model Fusion in Federated Learning](/papers/2006.07242): FedDF, distilling client ensembles into a server model. - [The Impact of Feature Scaling In Machine Learning](/papers/2506.08274): tree ensembles largely insensitive to the scaler. - [Application of Artificial Intelligence for Fraudulent Banking Operations Recognition](/papers/2608.07471): stacked ensemble on card-fraud data. - [Precision in Rice Variety Classification using Stacking-Based Ensemble Learning](/papers/2609.10524): stacked CNNs with an XGBoost meta-model. - [Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations](/papers/2605.25305): weaker-model booster over a tuned LSTM for 12-month campus electricity use.