Ensemble learning

Overview

An ensemble averages, votes or fuses several models so that the combined predictor beats its members. The cards cover four basic moves: train members on different samples (bagging), train each member to fix the previous one's errors (boosting), learn how to combine member outputs (stacking), or pick a member per input (dynamic selection, LLM ranking). Most cards are empirical. Each one reports gains for its own datasets and tuning budget, so the page records which conditions each gain was measured under.

Variants

Bagging and random forests

Members are trained on resampled rows and their outputs are averaged. Random forests are the usual tree form [[random-forest]]. Bagging also supplies the candidate pool for dynamic selection: one benchmark builds 100 bagged perceptrons and selects among them per query 2212.12343 §3.2 tab. 1. Card: Explaining a Series of Models by Propagating Shapley Values notes that a bagged or boosted ensemble is linear in its members, so it can be explained directly 2105.00108 app. A.5.4.

Boosting

Trees are added one at a time, each fitted to the errors of the ones before. The most used implementation in the cards is XGBoost, which the tabular benchmark found usually the best single model and the quickest to tune 2106.03253 §3.2 [[gradient-boosted-decision-trees]]. Card: Tabular Data: Deep Learning is Not All You Need. For class imbalance, a review lists RUSBoost, which combines random under-sampling with boosting, and says it is simpler and faster than SMOTEBoost. The review runs no experiments to check this 1305.1707 §3 [[adaboost]]. Card: Class Imbalance Problem in Data Mining Review.

Differentiable variants replace the hard trees with soft ones trained by gradient descent. NODE is one; it beats default CatBoost and XGBoost on all six of its datasets, and tuned versions on four of six, but trains and infers more slowly than the GBDT baselines 1909.06312 §4.1 tab. 4. Card: Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data.

Stacking and voting

A meta-model learns from the base models' outputs; voting uses a fixed rule. Three cards test these on imbalanced, fraud and image-feature data, with different sets and metrics. A review and benchmark shows Stacking-II above Stacking-I on its AUC heatmaps, and its best named pairs are gradient-boosting models with resampling 2304.02858 §5.3 2304.02858 §7. Card: Imbalance benchmark pairing augmentation with ensembles. A rice-variety study stacks four fine-tuned CNNs under an XGBoost meta-model and reports 1.000 accuracy on its own set 2609.10524 §4.4 tab. 6. Card: Precision in Rice Variety Classification using Stacking-Based Ensemble Learning. On card-fraud data, a stack reaches AUC 0.954 against 0.946 for the best single model, logistic regression 2608.07471 §5. Card: Application of Artificial Intelligence for Fraudulent Banking Operations Recognition. See [[stacked-generalization]] for the stacking details.

Ensembles of deep or mixed models on tabular data

Deep networks can be ensembled with each other or with a tree model. On 11 datasets, a validation-loss-weighted ensemble of four deep models plus XGBoost scored 2.32% mean relative deterioration (lower is better). XGBoost alone scored 3.34%, and the deep models without XGBoost 6.91% 2106.03253 §3.2 tab. 3. The weighting formula has a sign ambiguity, so reproduce it with care 2106.03253 §2.2 eq. 2.

Ensemble distillation in federated learning

FedDF replaces FedAvg's weight averaging with distillation. The server trains one model to match the averaged logits of the client models on unlabelled data 2006.07242 §3 eq. 1. It needed about 20 rounds on CIFAR-10 against 104 for FedAvg, with α = 1, 40 local epochs and 40% participation 2006.07242 tab. 1. The server still uses labelled validation data for early stopping 2006.07242 §4.1. Card: Ensemble Distillation for Robust Model Fusion in Federated Learning. See [[knowledge-distillation]] and [[federated-aggregation]].

Averaging over encoders, crops and folds

Frozen ImageNet encoders give fixed features; one LightGBM model is fitted per encoder, crop size, cross-validation fold and seed, and the predicted probabilities are averaged over augmentations, seeds and the six encoder/crop models 1802.00752 §2.5. On 400 histology images with 10-fold cross-validation, the author-reported four-class accuracy of the blend is 87.2 ± 2.6%. The six single models range from 82.5% to 84.2% mean accuracy, and the best of them, ResNet at 400 px, reaches 84.2 ± 4.2% [arxiv:1802.00752 tab. 1]. The two-class task is reported at 93.8% without a baseline 1802.00752 §3. Card: Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis. See [[transfer-learning]] and [[gradient-boosted-decision-trees]].

One network that acts as an ensemble

A batched ensemble of random, untrained representations can run as one network. Deep Isolation Forest stacks 50 representations with six isolation trees each, scoring points by path depth 2206.06602 §5.2.1 eq. 3-5. Its author-reported AUC-ROC of 0.925 on ten tabular sets is above the deep ensembles' 0.889 2206.06602 tab. 4. Card: Deep Isolation Forest for Anomaly Detection. See [[isolation-forest]].

Mixtures of models for Bayesian optimization

EGP-TS keeps one Gaussian process per kernel in a dictionary and weights them by fit. Each query is chosen by Thompson sampling from the weighted mixture, using 50 random features per kernel 2205.14090 §3 2205.14090 §3.1. The author-reported regret is lower than single-GP Thompson sampling and two bandit ensembles fig. 1–3. Card: Surrogate modeling for Bayesian optimization beyond a single Gaussian process. See [[bayesian-optimization]].

Fusing LLM outputs

LLM-Blender ranks N candidate answers pairwise, then fuses the top three into one answer. On MixInstruct it reaches GPT-Rank 3.01 against 3.90 for the best single model 2306.02561 §2.3 tab. 2. The judge model is a ChatGPT version the paper does not name, so the rank is not reproducible from the paper alone. The released code does more pairwise passes than the paper states; plan the budget with the code's count 2306.02561 §3.3. Card: LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. See [[llm-as-a-judge]].

Hybrid classifiers for time series

HIVE-COTE 2.0 (HC2) trains four modules on the same training set: the Shapelet Transform Classifier (STC), the Temporal Dictionary Ensemble (TDE), the Diverse Representation Canonical Interval Forest (DrCIF) and Arsenal, an ensemble of ROCKET-style classifiers 2104.07551 §3 §3.1 §3.2 §3.3 §3.4. Each module estimates its own accuracy from the training data. A new case gets a weighted sum of module probabilities, with each accuracy estimate raised to the power 4 2104.07551 §3. The paper gives no formula for the combination. Card: HIVE-COTE 2.0: a new meta ensemble for time series classification.

Author-reported results, where a negative mean difference means HC2 is better. On 112 univariate UCR problems the mean difference is −1.06% against HC1, −1.36% against TS-CHIEF, −1.69% against InceptionTime and −2.49% against ROCKET 2104.07551 tab. 3. On 26 multivariate UEA problems the largest gap is −8.22% against DTW-D 2104.07551 tab. 5. The cost is run time: one sequential UCR resample takes about 340 hours for HC2, against about 2.85 hours for ROCKET 2104.07551 tab. 4. On the five problems where a sequential build takes over 12 hours, a 4-hour time contract reaches 98% of the full-build accuracy and a 12-hour contract 99% 2104.07551 §7 tab. 8. An ablation over 11 variants built from the four modules finds the full ensemble significantly better than each of them 2104.07551 §6 tab. 7.

The Bake off redux re-run also ranks HC2 first on accuracy over 112 UCR problems, at 0.891, with MR-Hydra second at 0.884 2304.13029 §5 tab. 14. Its HC2 timing covers a different set of datasets, 263.89 hours in total over 142 problems 2304.13029 tab. 19, so it is not a direct check on the 340-hour figure above. The authors suggest HC2 and MR-Hydra as defaults without prior knowledge 2304.13029 §7. Card: Bake off redux. See [[time-series-classification]].

Ensemble models in a forecasting library

Darts puts its ensemble models behind the same fit and predict calls as its single forecasters, and one of them learns how to combine its members with a regression model 2110.03224 §2.6. The paper reports no accuracy benchmark for any model, so it gives no evidence on how these ensembles compare with their members 2110.03224 §3. Card: Darts: User-Friendly Modern Machine Learning for Time Series. See [[time-series-forecasting]].

Pushing a strong forecaster away from weaker ones

The Weaker Separator Booster takes a tuned LSTM as the strong member and the mean of three weaker members (random forest, SVR and XGBoost) as a reference. It adds to the strong forecast a term proportional to the gap between the two, scaled by a weight that is zero at the first step of a 12-month horizon and peaks in the middle 2605.25305 §3.2.3 eq. 1-3. On recursive 12-month tests of two campus electricity series, the author-reported MAE is 1990.87 kWh against 2264.50 for the LSTM alone at Palmas, and 464.93 against 466.92 at Coronel Vivida tab. 14 tab. 15. The Coronel gain is under 0.5%, and each campus was tested on one year only. The global weight 0.0833 came from a random search over 0 to 1 2605.25305 §3.2.3. Card: Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations. See [[time-series-forecasting]].

When to use which

Implementation sketch

Common core: fit members, combine their outputs. Stacking adds a second fit. The cards disagree on how the meta-model's training inputs are made.

# Fit each base model on the training split  [arxiv:2304.02858 §5.1]
for m in base_models:
    m.fit(X_train, y_train)

# Soft voting: average the class probabilities  [arxiv:2304.02858 §4.1]
def soft_vote(models, x):
    return mean(m.predict_proba(x) for m in models)

# Stacking: the meta-model learns from base outputs  [arxiv:2609.10524 §4.4]
# Open choice: out-of-fold or in-sample base outputs. Paper does not say.
Z_train = stack_outputs(base_models, X_train)  # shape: rows x (models*classes)
meta.fit(Z_train, y_train)

# Predict: base outputs, then meta-model  [arxiv:2609.10524 §4.2]
def stack_predict(x):
    return meta.predict(stack_outputs(base_models, x))

Contradictions

• Deep networks vs trees. The NODE paper reports wins over tuned XGBoost on four of six datasets 1909.06312 §4.1. The 11-dataset benchmark reports XGBoost ahead on 8 of 11 datasets outside the deep models' own papers 2106.03253 §3.2. Different datasets and tuning protocols. Neither is resolved here.
• Voting rule. The benchmark text says hard voting is generally better than soft voting 2304.02858 §5.3. Its released code builds only soft voting models; see the Paper vs code section of its card.
• Stacking gains. A 1.000 accuracy on a self-curated rice set 2609.10524 tab. 6 and a 0.008 AUC gain on card-fraud data 2608.07471 §5 are not comparable: different tasks, metrics and baselines.
• Fusion gain in one paper. The histology paper says the fused model is 4 to 5 points better than any constituent 1802.00752 §3. Its Table 1 gives 87.2% against 84.2% for the best single model, a gap of 3.0 points 1802.00752 tab. 1. The two statements are not reconciled here.
• Within one paper. The deep-model benchmark places the simple ensemble ahead of XGBoost in its table, but the text calls XGBoost second best 2106.03253 tab. 3 §3.2.
• Booster result in one paper. The Coronel Vivida MAE of the booster is 464.93 kWh in Table 15 and 465.02 kWh in the abstract 2605.25305 tab. 15 2605.25305 abstract. The difference is small, but the two values should be checked against each other before quoting either.
• Large-dataset runs in HC2. The text says which HC2 components finished on FruitFlies and InsectSound, and the result tables report accuracies for different components 2104.07551 §7 tab. 10 tab. 11. Use those numbers only after checking the tables against the text.

Papers

• Imbalance benchmark pairing augmentation with ensembles: stacking, voting and boosting with resampling, benchmarked on 27 sets.
• Class Imbalance Problem in Data Mining Review: survey of imbalance fixes, including boosting variants; no experiments.
• Deep Convolutional Neural Networks for Breast Cancer Histology Image Analysis: LightGBM over frozen CNN features, averaged over encoders, crops, folds and seeds.
• Darts: User-Friendly Modern Machine Learning for Time Series: forecasting library with ensemble models; no accuracy benchmark.
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data: differentiable oblivious-tree ensemble for tabular data.
• Explaining a Series of Models by Propagating Shapley Values: attributions through stacked and ensembled models.
• Tabular Data: Deep Learning is Not All You Need: deep-model and XGBoost ensembles on 11 tabular datasets.
• Deep Isolation Forest for Anomaly Detection: batched ensemble of random representations, one network.
• The choice of scaling technique matters for classification performance: scaler sensitivity of bagging, dynamic selection and other ensembles.
• Surrogate modeling for Bayesian optimization beyond a single Gaussian process: ensemble of Gaussian processes as a surrogate.
• Bake off redux: hybrid ensemble HC2 among time-series classifiers.
• HIVE-COTE 2.0: a new meta ensemble for time series classification: four-module meta ensemble (STC, TDE, DrCIF, Arsenal) on UCR and UEA problems.
• LLM-Blender: pairwise ranking and fusion of LLM answers.
• Ensemble Distillation for Robust Model Fusion in Federated Learning: FedDF, distilling client ensembles into a server model.
• The Impact of Feature Scaling In Machine Learning: tree ensembles largely insensitive to the scaler.
• Application of Artificial Intelligence for Fraudulent Banking Operations Recognition: stacked ensemble on card-fraud data.
• Precision in Rice Variety Classification using Stacking-Based Ensemble Learning: stacked CNNs with an XGBoost meta-model.
• Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations: weaker-model booster over a tuned LSTM for 12-month campus electricity use.