A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation

In plain words

Many real data sets have one class that is far rarer than the others, such as fraud, disease or intrusion cases. A standard classifier learns to favour the common class, so it can look accurate overall while missing the rare cases that matter most.

Two families of fixes are common. One adds synthetic copies of the rare cases (or removes common ones) before training. The other trains several models and combines their votes. This paper tries many pairs of these two families on the same benchmark data sets and reports which pairs did best. It is a comparison, not a new method. Its main message is that the best pair depends on the data set, and that the simplest oversampling methods hold up well against the newer GAN-based one.

TL;DR

• A review plus a computational benchmark. It evaluates data augmentation methods combined with ensemble learning models for class imbalance §1 §4.
• Every pair is run 30 times on a 60:40 train/test split of the chosen data sets. Reported metrics are accuracy, F1 and an AUC §5.1 §4.2.
• Author-reported best pairs in the conclusion: SMOTE + LightGBM and ROS + LightGBM. Traditional SMOTE and ROS are said to beat CT-GAN and to cost less §7.
• The per-dataset tables do not always agree with the text or the conclusion. The code also differs from the protocol in several places. See Pitfalls and Paper vs code.

Problem

In a skewed data set one class has many more samples than the other §1. Most classifiers are biased toward the majority class and can predict only the majority class while ignoring the minority class §1. The paper uses medical diagnosis as its example of a model that cannot reliably identify the rare cases §1.

Accuracy is a poor guide here. The paper's example: at an imbalance ratio of 9.5, an accuracy of 95 % can mean every minority sample was misclassified §4.2. The paper defines the imbalance ratio (IR) as the number of majority instances divided by the number of minority instances §4.2. It gives no threshold at which imbalance starts to matter §1.

Method

The paper has two parts. Sections 2 and 3 review ensemble learning and data augmentation. Section 4 defines the benchmark, and section 5 runs it §1.

1. Data sets §5.2 tab. 2. The binary sets are derived from three UCI sets (Glass, Yeast, Ecoli) by merging some classes into a positive (minority) class and the rest into a negative class. The naming follows Galar et al., so "Ecoli-0137vs26" means classes 0, 1, 3 and 7 are positive and 2 and 6 are negative §5.2. The 23 binary sets range from IR 1.82 to IR 41.38 tab. 2. Multi-class sets come from Abalone, whose rings-based age is grouped into classes; four versions with three or four classes are used §5.4 tab. 3.

2. Data augmentation §4.1 tab. 1. The framework includes nine methods: SMOTE, SMOTE-ENN, Borderline-SMOTE, SVM-SMOTE, K-means SMOTE, ADASYN, random under-sampling (RUS), random over-sampling (ROS) and CT-GAN tab. 1 main.py. The SMOTE family creates new minority samples; the paper calls SMOTE the most prominent approach for CI data §3.1. CT-GAN is the GAN-based method §3.3. The text of §4.1 names only eight methods and omits K-means SMOTE, which appears in tab. 1.

3. Ensemble models §4.1 fig. 2 fig. 3.

1. Bagging and boosting models: LightGBM, XGBoost, AdaBoost, Gradient Boosting and Random Forest §2 tab. 1.
2. Voting models. The paper names Voting-Soft and Voting-Hard §4.1. Its figure describes two voting architectures: Voting-I combines a decision tree, a random forest, a k-nearest-neighbours model and XGBoost, and Voting-II combines a random forest and XGBoost fig. 3. The paper does not say which voting rule each architecture uses. See Paper vs code.
3. Stacking models. Stacking-I uses the same four base learners as Voting-I. Stacking-II uses a random forest and XGBoost. Both use logistic regression as the meta-learner §4.1 fig. 2.

The text says the framework uses "ten" ensemble methods but lists nine §4.1. The AdaBoost, gradient boosting and XGBoost formulas are in §2 and are summarised here only by reference: AdaBoost weights eq. 1–eq. 4, gradient boosting eq. 5–eq. 11, XGBoost eq. 12–eq. 17.

4. Protocol §5.1 fig. 4. Each pair of one augmentation (or none) and one ensemble is run on each data set. The paper reports 100 such combinations per data set, which implies ten ensemble settings times ten augmentation settings including "no augmentation" §5.3. Each run augments the training split, fits the model, and predicts the test split. The paper reports the mean, best and standard deviation over 30 runs §5.1 §5.3.

5. Metrics §4.2. Accuracy, F1 and AUC are reported for binary sets. Precision and recall are defined but not tabulated. For multi-class sets the paper reports F1 macro, F1 weighted and AUC with one-vs-one and one-vs-rest §4.2.1.

6. Decision threshold §4.2.2. Hard labels use the scikit-learn default probability threshold of 0.5 §4.2.2.

# Benchmark protocol for one (augmenter, ensemble) pair  [§5.1] [fig. 4]
# Input:  features X (N, d), labels y in {0, 1} (1 = minority)
# Output: mean, best and std of accuracy, F1 and AUC over 30 runs

def evaluate_pair(X, y, augmenter, ensemble):
    # Split 60:40 into training and test rows  [§5.1] [code: main.py#L145]
    X_train, X_test, y_train, y_test = split(X, y, test_size=0.4)
    results = []
    # Repeat 30 times  [§5.1] [§5.3]
    for run in range(30):
        # Augment only the training rows, inside each run  [§5.1]
        X_aug, y_aug = augmenter.resample(X_train, y_train)
        # Fit the ensemble on the augmented rows  [§5.1]
        ensemble.fit(X_aug, y_aug)
        # Hard labels at the default 0.5 threshold  [§4.2.2]
        y_pred = ensemble.predict(X_test)
        # Accuracy: [eq. 18]
        acc = accuracy(y_test, y_pred)
        # F1: harmonic mean of precision and recall  [eq. 24]
        f1 = f1_score(y_test, y_pred)
        # AUC by the paper's single-threshold formula  [eq. 21]
        # The code applies it to hard labels; see Paper vs code
        auc = (1 + tpr(y_test, y_pred) - fpr(y_test, y_pred)) / 2
        results.append((acc, f1, auc))
    # Report mean, best and standard deviation  [§5.3]
    return summarise(results)

The paper does not say which random seeds were used §5.1. The code sets one seed, for the split only main.py.

The metric equations, as the paper writes them eq. 18–eq. 24:

Accuracy = \frac{TP + TN}{TP + FN + FP + TN} eq. 18

TPR = \frac{TP}{TP + FN} eq. 19

FPR = \frac{FP}{FP + TN} eq. 20

AUC = \frac{1 + TPR - FPR}{2} eq. 21

precision = \frac{TP}{TP + FP} eq. 22

recall = \frac{TP}{TP + FN} eq. 23

F1 = \frac{2 \cdot precision \cdot recall}{precision + recall} eq. 24

Hyperparameters

The authors state that they use the default hyperparameters of the scikit-learn library for the ensemble and augmentation methods §5.1. The values in tab. 1 are those defaults as the paper lists them. The code sets almost none of them: only kind='borderline-1' for Borderline-SMOTE and sampling_strategy='auto' for K-means SMOTE main.py main.py. Everything else comes from the library versions installed at run time, which the repository does not pin main.py.

Datasets

Binary sets, from tab. 2. IR is majority count divided by minority count.

The multi-class Abalone versions are in tab. 3: Abalone-v1 has four classes (189, 2577, 1186, 225 samples); v2, v3 and v4 have three classes each.

Results

Author-reported. Setup: 60:40 split, 30 runs per pair, default scikit-learn hyperparameters, 23 binary and 4 multi-class sets §5.1 tab. 2 tab. 3. Each dataset's table shows 11 of the 100 combinations, from the 100th to the 0th percentile by accuracy §5.3.

Best and worst combination per binary dataset, as the authors summarise them tab. 8:

Example rows from the per-dataset tables, each cell as mean (best, std) over 30 runs:

Text claims without the per-dataset tables:

• F1 rankings over the 30-run means, with LightGBM as the fixed model for the augmentation ranking. The text says SMOTE-ENN is best on Glass, SMOTE and Borderline-SMOTE on Yeast, and ADASYN on Ecoli. RUS is worst on all three §5.3 fig. 5 fig. 6 fig. 7.
• AUC heatmaps show Stacking-II above Stacking-I, and Voting-Hard generally above Voting-Soft §5.3.
• Multi-class (Abalone) F1 macro: SMOTE + Stacking-I and SMOTE + LightGBM rank top; SMOTE-ENN + AdaBoost ranks worst. For F1 weighted, models without augmentation rank top, with Gradient Boosting first §5.4 tab. 9 tab. 10.
• The paper concludes that the combinations SMOTE + LightGBM and ROS + LightGBM are the best and that CT-GAN is not a top performer §7.

Paper vs code

Repository: sydney-machine-learning/review-classimbalanced-ensemblelearning, linked from the paper's "Code and Data" section §7. The owner is the authors' lab. It has no LICENSE file, so no reuse licence is stated. The card cites code by path and line only, and quotes none.

• Split. The paper says the data are shuffled randomly for each experiment §5.1. It also says the train/test split is fixed for each run and only initial weights are re-drawn §5.3. The code uses one split, train_test_split(..., test_size=0.4, random_state=42), without stratification main.py. The code matches §5.3, not §5.1.
• Augmentation runs. The paper augments inside each run §5.1. The code calls sm.fit_resample(X_train, y_train) inside the 30-run loop and writes the result back into X_train main.py. Each run therefore resamples the previous run's output. For samplers that generate nothing when the classes are already balanced (the imbalanced-learn default for sampling_strategy='auto', not run here), this changes nothing after the first run. For SMOTE-ENN, whose cleaning step can unbalance the data again, the training set changes from run to run. The run-to-run statistics for such pairs are therefore not what the paper describes.
• No-augmentation runs. The code fits the same model 30 times on identical data main.py. Deterministic models give identical results, so their standard deviation is 0. Rows with std 0.000 in tab. 4 and tab. 5 fit this pattern, but the repository does not show it directly. Models with no fixed seed (random forest, XGBoost) do vary.
• AUC. The paper defines AUC as the area under the ROC curve over thresholds §4.2 and gives the single-threshold formula eq. 21. The code calls roc_auc_score(y_test, y_pred) on hard predictions main.py main.py. For binary labels that equals (1 + TPR − FPR) / 2, that is, balanced accuracy. No threshold sweep is computed.
• Voting-Hard. The results report "Voting-Hard" and say it is generally better than "Voting-Soft" tab. 4 §5.3. The code builds only soft-voting models: VCI and VCII both use voting='soft' main.py main.py. The code cannot reproduce the Voting-Hard rows.
• Decision tree. The code also offers a plain decision tree model main.py. The paper's framework and tables do not include it.
• CT-GAN. The notebook imports tabgan.sampler.GANGenerator CT-GAN.ipynb and calls it with gen_x_times=1.1, quantile filters 0.001 and 0.999, batch_size 500, patience 25 and epochs 500 CT-GAN.ipynb. The adversarial model settings match the tab. 1 CT-GAN row. This is a third-party library, not the original CT-GAN; the paper does not name the library. The paper's network description (50, 25, 10 hidden nodes, MSE, Adam, 1000 epochs §5.1) does not match the notebook's Keras cell, which has 50, 25, 12 hidden nodes, a single output, EarlyStopping with patience 5 and up to 1000 epochs CT-GAN.ipynb. It is not clear which network the paper's CT-GAN row used.
• Scaling. StandardScaler and MinMaxScaler are imported but never applied main.py main.py. The paper does not mention feature scaling.
• Hyperparameters. The code sets almost none, so the other tab. 1 values are library defaults as the paper states them. K-means SMOTE is built with its defaults main.py, which the paper lists as k_neighbors = 2. The library version is not pinned, so the match cannot be checked.
• Multi-class. The repository contains Datasets/Abalone Datasets/AbaloneMIF.csv, but main.py offers no Abalone option and has no multi-class metric code main.py. The multi-class results cannot be reproduced from this repository.
• Data loading. main.py reads the CSVs from raw.githubusercontent.com on the main branch main.py, not from a pinned commit. The code_commit above is the version read here.

Pitfalls

• Counts disagree. The abstract says 9 augmentation and 9 ensemble methods. §4.1 says "nine" augmentation methods but lists eight, and says "ten" ensemble methods but lists nine §4.1. The "100 combinations" §5.3 implies ten of each, including "none".
• Abalone classes. The text describes four age classes (0–5, 6–10, 11–15, 16–20 years) and comments out a fifth §5.4. tab. 3 lists four-class and three-class versions, and the text does not map each version to its classes. The text calls Abalone a regression problem before it is grouped.
• Text and conclusion disagree. §5.3 names SMOTE-ENN as the best augmentation on Glass, but the conclusion names SMOTE + LightGBM as best. The text names CT-GAN + LightGBM as best on Ecoli, but tab. 8 lists Stacking-1 + CT-GAN as best on Ecoli-0vs1 and Ecoli-3. The conclusion says CT-GAN is not a top performer, but tab. 8 lists CT-GAN among the best pairs on five of the 23 binary sets.
• Split sizes. The split is not stratified. For Yeast-2vs8, with 20 minority rows in 483 tab. 2, a 40 % test set holds about 8 minority rows. Each error on so few rows moves minority recall a lot. This may explain the high standard deviations in tab. 4, such as Glass-6 with ROS + AdaBoost (F1 std 27.641), but the paper does not test it.
• "Best" columns. The table reports the best of 30 runs next to the mean. A best value of 100.000 means one run was perfect, not that the pair is reliable.
• Std 0.000 rows. Zero standard deviation is expected for deterministic models under this protocol (see Paper vs code), so it is not evidence of stability.
• Cost claim. The abstract and conclusion say SMOTE and ROS are cheaper than GANs §7. The paper reports no timing or memory numbers, so this claim is not backed by data in the paper.
• Accuracy as a headline. The paper itself warns that accuracy misleads on imbalanced data §4.2. Several per-dataset rows still rank by accuracy §5.3 tab. 4. Compare by F1 or AUC.

Lineage

• Builds on: the individual methods it benchmarks (SMOTE, ADASYN, AdaBoost, gradient boosting, XGBoost, LightGBM, stacking), each cited in §2 and §3. No single earlier method is extended.
• Compared against: the "no augmentation" baseline for each ensemble, and the other augmentation methods within the framework §5.3.
• Superseded by: not stated.