Stacked generalization

Overview

Stacked generalization combines several models with a second, trained model. The base models make their predictions first. A meta-model then takes those predictions as its inputs and learns how to combine them into one answer 2609.10524 §4.4 2608.07471 §3.2 2304.02858 §4.1. The meta-model is trained on the base models' outputs, not on the raw data alone.

The wiki has four cards on this family so far. Three use a trained meta-model: one on image classification, one on tabular fraud detection and one on class-imbalance benchmarks that include stacks. The fourth, on time-series classification, is a close relative. It combines its members with fixed weights and trains no meta-model 2104.07551 §3. The four use different base models, data and metrics, so their numbers are not comparable.

In the three trained-meta-model papers, the paper does not state whether the meta-model is trained on out-of-fold base outputs. The image paper also does not say which split its headline figure uses 2609.10524 §4.4 2609.10524 tab. 6. Those are the details that decide whether a stacked result can be trusted, so an implementation has to choose and state them itself. The fixed-weight paper avoids the question differently: its weights come from accuracy estimates built on separate bagged fits or out-of-bag samples, not from the training fit itself 2104.07551 §3 2104.07551 §6.

Variants

Stacked CNNs with an XGBoost meta-learner

Ten ImageNet-pretrained CNNs are fine-tuned on the same 20-class image set. The four best are kept: EfficientNetV2L, VGG16, MobileNetV2 and DenseNet201 2609.10524 §4.2 2609.10524 §5. Their outputs feed a meta-model. The authors compare 14 meta-learners and choose XGBoost 2609.10524 §4.4 2609.10524 tab. 6. Card: Precision in Rice Variety Classification using Stacking-Based Ensemble Learning.

Reported result: with the top four base models and XGBoost, accuracy is 1.000 on the authors' own set 2609.10524 tab. 6. The base models reach 0.9747 to 0.9924 validation accuracy 2609.10524 tab. 4. On a separate external set, the paper reports 698 of 700 correct for one of five varieties, and the text does not name the model behind that result 2609.10524 §5.

Decision tree, random forest and extra trees also reach 1.000 on the same combination 2609.10524 tab. 6. The paper does not explain why XGBoost is preferred over them.

Stacked classifiers on imbalanced tabular data

Seven standard classifiers are trained on the Kaggle card-fraud data: random forest, k-nearest neighbours, logistic regression, SGD, decision tree, naive Bayes and SVM 2608.07471 §3.1. The training split is balanced by random undersampling, and only the Time and Amount columns are standardized 2608.07471 §3.4. Card: Application of Artificial Intelligence for Fraudulent Banking Operations Recognition.

The stacked variant trains K weak classifiers, each on a random subset of size B, and combines their outputs with a meta-model 2608.07471 §3.2 2608.07471 eq. 2. The paper does not give K, B, the base classifiers or the meta-model, so this variant cannot be rebuilt from the paper alone.

Reported result: the stacked ensemble reaches AUC 0.954, against 0.946 for the best single model, logistic regression 2608.07471 §5 2608.07471 fig. 5. Its F1 of 0.96 is the highest the paper reports 2608.07471 §4.2 2608.07471 fig. 6. The other single models score from 0.908 to 0.938 AUC 2608.07471 §4.2 2608.07471 fig. 4. The authors conclude that all seven models "more or less equally cope" 2608.07471 §5. The gap is about 0.008 AUC, from one run on one dataset, with no variance reported.

Stacking-I and Stacking-II in a resampling benchmark

The review pairs nine resampling methods with nine ensemble models, two of them stacks, on 23 binary and four multi-class data sets 2304.02858 §4.1 2304.02858 §5.1. Stacking-I uses decision trees, random forests, k-nearest neighbours and XGBoost as base models. Stacking-II uses random forests and XGBoost. Both use logistic regression as the meta-learner 2304.02858 §4.1 2304.02858 fig. 2. Card: Khan et al., ensemble and augmentation benchmark.

Reported result: in the paper's per-set summary, Stacking-I is the best pair on four of the 16 binary sets listed. Those are Ecoli-0vs1, Ecoli-3, Glass-5vs12 and Yeast-2vs8, paired with CT-GAN or SVM-SMOTE 2304.02858 tab. 8. Stacking-II is not the best pair on any listed set, and it is the worst pair on three of them 2304.02858 tab. 8. Each pair is run 30 times on a 60:40 split, and results are reported as means 2304.02858 §5.1 2304.02858 §5.3.

The text says Stacking-II beats Stacking-I on AUC heatmaps for six data sets 2304.02858 §5.3. This is a different summary from the best-pair table, and the review does not reconcile the two.

Accuracy-weighted meta-ensemble, without a meta-model

HIVE-COTE 2.0 trains four time-series classifiers on the same training set: the Shapelet Transform Classifier, the Temporal Dictionary Ensemble, the Diverse Representation Canonical Interval Forest and Arsenal 2104.07551 §3. Each one gives class probabilities for a new series, plus an estimate of its own accuracy from the training data. The probabilities are combined with weights taken from those estimates, raised to the power 4 2104.07551 §3. No second model is trained. The paper does not write the combination out as a formula; the form on this page is our reading of §3. Card: HIVE-COTE 2.0: a new meta ensemble for time series classification.

Reported result: on the 112 univariate UCR problems, the full system is significantly more accurate than its four components and than the four benchmark methods 2104.07551 §5.1 2104.07551 fig. 6. Its own ensembling comparison puts the accuracy-weighted average, called CAWPE, ahead of the alternatives it tested. These include choosing the best member on the training estimates, forward selection, ensemble selection and random-forest stacking 2104.07551 §6.1 2104.07551 fig. 18. This is the paper's own comparison, on its own benchmark.

The cost is compute. A sequential build of one UCR resample takes about 340 hours, against about 2.9 hours for ROCKET 2104.07551 tab. 4. A time contract caps training at some cost in accuracy 2104.07551 §7.

When to use which

The evidence is four papers on four different tasks, so each row is tied to the setting it comes from.

Implementation sketch

The core is the same for any base models and meta-model. Only the base architectures and the meta-learner change. The open choice is where the meta-model's training inputs come from. Whether they are out-of-fold is the decision the three trained-meta-model papers leave unstated.

The fraud paper's subset variant differs in one step. Each base classifier is trained on its own random subset, rather than on the full training set 2608.07471 §3.2 2608.07471 eq. 2. The subset count and size must be chosen by the implementer.

The review resamples the training rows inside each run, before fitting, and leaves the test rows alone 2304.02858 §5.1.

HIVE-COTE 2.0 uses the same loop with one change: the meta-model step is replaced by a weighted average. Each base model's weight is its accuracy estimate from the training data, raised to the power 4. The estimate comes from a separate bagged build, or from out-of-bag samples, rather than from one fit on all rows 2104.07551 §3 2104.07551 §6.

# Stacked generalization: a meta-model learns to combine base models
# Input:  train_x, train_y, test_x
# Output: predicted label for each test row

def stacked_fit_predict(train_x, train_y, test_x, base_specs, meta_spec):
    # Optional for imbalanced labels: resample the training rows only [arxiv:2304.02858 §5.1]
    train_x, train_y = resample(train_x, train_y)
    # Fit each base model on the training data [arxiv:2609.10524 §4.2]
    base_models = [fit(spec, train_x, train_y) for spec in base_specs]
    # Turn each base model's outputs into meta-features [arxiv:2609.10524 §4.4]
    # Out-of-fold or in-sample training outputs: not stated in the three trained-meta-model papers
    meta_train = stack_outputs(base_models, train_x)
    meta_test = stack_outputs(base_models, test_x)
    # Fit the meta-model on the stacked outputs [arxiv:2609.10524 §4.4]
    # The review's meta_spec is logistic regression [arxiv:2304.02858 §4.1]
    meta_model = fit(meta_spec, meta_train, train_y)
    return meta_model.predict(meta_test)

Contradictions

The four papers do not disagree on a shared result. Their numbers come from different tasks and metrics, so they cannot be compared. Each card also has internal inconsistencies, listed below.

Image card 2609.10524:

• Stacking inputs. The text describes the meta-model as taking predicted outputs 2609.10524 §4.4. Section 4.2 and fig. 3 describe stacking "feature sets" 2609.10524 §4.2 2609.10524 fig. 3.
• Cross-validation for the meta-model. The text says the meta-model is trained "using cross-validation", and gives no folds for it 2609.10524 §4.4.
• Split behind the headline figure. The text calls the 1.000 figure "100% test accuracy", while table 4 is labelled validation 2609.10524 §5 2609.10524 tab. 4. Table 6 does not say which split it uses 2609.10524 tab. 6.

Fraud card 2608.07471:

• Balancing. Section 3.4 says undersampling was used "as the dataset was not imbalanced" 2608.07471 §3.4. Section 4.1 calls the same data "highly unbalanced" 2608.07471 §4.1.
• Selection split. The workflow selects the model "with the best evaluation metric on the testing data" 2608.07471 fig. 3. Section 4.2 describes a separate validation split 2608.07471 §4.2.
• Gain size. The conclusion calls the 0.008 difference "an AUC of 0.008" 2608.07471 §5. It is the gap between 0.954 and 0.946, not an AUC.

Review card 2304.02858:

• AUC. The paper defines AUC with a single-threshold formula 2304.02858 eq. 21. The card records that the authors' code applies its ROC-AUC function to hard predictions, which gives balanced accuracy. The stacked "AUC" values in its tables are therefore not threshold-based AUCs. See the card's Paper vs code section.
• Stacking-II versus Stacking-I. The AUC heatmaps favour Stacking-II 2304.02858 §5.3, while the best-pair table lists Stacking-I four times and Stacking-II never 2304.02858 tab. 8.
• Split size. The split is not stratified. On Yeast-2vs8, a 40 % test set holds only about eight minority rows 2304.02858 tab. 2, so a single error moves minority recall a lot. The paper does not test whether this explains its high standard deviations.

HIVE-COTE 2.0 card 2104.07551:

• Combination rule. The paper describes the tilt and the accuracy weights in words and gives no formula 2104.07551 §3. The form in the implementation sketch is our reading.
• Voting in the code. The text says Arsenal classifies new cases by majority vote 2104.07551 §3.3. The card records that the authors' code weights each Arsenal member by its ridge cross-validation score, raised to the power 4 2104.07551 §3.3. The paper does not name the voting scheme used to combine the four modules.
• Fixed weights against a learned stack. The paper reports that its weighted average beats random-forest stacking on its benchmark 2104.07551 §6.1. The three trained-meta-model cards above do not test a fixed-weight baseline, so that result is not checked in their settings.

Papers

• Precision in Rice Variety Classification using Stacking-Based Ensemble Learning: stacks four fine-tuned CNNs under 14 meta-learners, with XGBoost chosen as the meta-learner. Author-reported, and the split details are unclear.
• Application of Artificial Intelligence for Fraudulent Banking Operations Recognition: stacks seven classifiers' outputs on card-fraud data and reports AUC 0.954 against 0.946 for logistic regression. Author-reported, on one dataset, with the stacking details not stated.
• Khan et al., ensemble and augmentation benchmark: pairs two stacks (Stacking-I and Stacking-II, logistic regression as meta-learner) with resampling methods on 27 imbalanced sets. Author-reported, with an AUC that the code computes from hard labels.
• HIVE-COTE 2.0: a new meta ensemble for time series classification: combines four time-series ensembles with accuracy-based weights and no meta-model. Compared against random-forest stacking on 112 UCR problems. Author-reported, and about 340 sequential hours per UCR resample.