Effective Class-Imbalance learning based on SMOTE and Convolutional Neural Networks

In plain words

In many classification datasets one answer is much rarer than the others, for example a disease or a fraud. A model trained on such data tends to favour the common answer, so its headline score can look good while it misses the rare cases. This paper tries to fix that by changing the training data before a neural network sees it.

The authors compare six ways to change the data. Some remove common examples, some copy rare ones, and SMOTE builds new rare examples by mixing neighbouring rare examples. Every method is followed by rescaling the features to a fixed range. Each treated dataset is then fed to one of two small networks, a plain fully connected network or a network with one-dimensional convolutions, trained with a loss function that gives less weight to examples the model already gets right.

The experiments use 24 imbalanced datasets. Each one is reshuffled and split 100 times, so the reported numbers are averages over those runs. The best combination in the paper is SMOTE, rescaling and the convolutional network. The authors report about 99% average accuracy for it, with small gaps between the methods. The paper does not report run-to-run variation, and the card's pitfalls cover several reasons to read the numbers with care.

TL;DR

Empirical study with a small architecture, not a new resampling method. The pipeline is: resample the data, normalise it, split it, train a DNN or 1D CNN with focal loss, and average eight metrics over 100 shuffles fig. 2 §3 §4.5. SMOTE with normalisation and the CNN has the highest average on all eight metrics, at 99.08% accuracy tab. 11. The next-best average accuracy is 99.01% (NearMiss with the DNN), and the lowest is 98.22% (RUS with the DNN) tab. 11 tab. 10. Preprocessing order and validation are not fully described (see Pitfalls).

Problem

A classifier trained on a set where one class heavily outnumbers the other tends to favour the majority class, and standard losses and accuracy do not flag the failure §1 fig. 1. The authors list two families of fixes: resampling (oversampling the minority class, undersampling the majority class) and cost-sensitive or ensemble methods, and state that undersampling can lose useful data while oversampling can overfit §1 §2. They argue that deep networks, especially CNNs, have not been studied enough in combination with resampling §1.

Method

Preprocessing, as drawn in the paper's pipeline, runs in this order: resampling with one of six methods, normalisation, a random split into training and test sets, training of one classifier, evaluation, and comparison across pipelines fig. 2 §3.

1. Resampling methods (applied one at a time, six pipelines in total).

• Random over-sampling (ROS). Minority rows are copied at random until the classes are closer to balanced §3.1.1 ROS fig. 3. The text says the majority class is increased, which contradicts the rest of the paragraph; see Pitfalls.
• SMOTE. Synthetic minority rows are created by interpolation between a minority sample and one of its neighbours. The paper gives three steps: pick a minority sample at random, take its K nearest minority neighbours (default K = 5), and pick N of those neighbours at random for interpolation, where N is the number of new rows needed to reach a 1:1 ratio §3.1.1 SMOTE fig. 4.
• Random under-sampling (RUS). Majority rows are removed at random until the data is more balanced; the paper notes this discards information §3.1.2 RUS fig. 5.
• Tomek links (TL). Two samples from different classes form a link when their distance satisfies the inequalities in eq. 1. The majority member of each link is removed §3.1.2 Tomek Links fig. 6. The text describes the link as holding when "at least one" of the two inequalities holds, which is looser than a mutual-nearest-pair reading; see Pitfalls.
• One-sided selection (OSS). Combines Tomek links with a condensed nearest neighbour rule to remove borderline and redundant majority samples; the paper gives no parameters beyond its description §3.1.2 OSS fig. 7.
• NearMiss. Removes majority samples chosen by their mean distance to minority samples. Three variants are described, and the paper reports NearMiss-2 as the strongest in the cited work. The default is three neighbours [§3.1.2 NearMiss] fig. 8. Which variant the experiments use is not stated.

2. Normalisation. Features are scaled so that their values lie between zero and one §3.1.2 Normalization. The method (for example min-max) and whether the scale is fitted on training data only are not stated.

3. Split and repetition. Data is shuffled and split into training and test sets; the split ratio is not stated. Each run uses a new shuffle and a new split, and this is repeated 100 times per dataset and pipeline §3.1.3 §4.2.

4. Models. Two networks are compared.

• DNN. Linear layer (64 units), ReLU, batch normalisation, linear (64), ReLU, batch normalisation, dropout 0.3, and a single-unit linear output tab. 2 fig. 9.
• CNN. Two one-dimensional convolutions, each followed by ReLU, then two linear layers with ReLU after the first. The convolution hyperparameters are in tab. 3 and the layer order in fig. 10. The layer dimensions in the table do not all match each other; see Pitfalls.

5. Loss. Focal loss. The paper first writes binary cross-entropy eq. 2, rewrites it with the probability of the true class eq. 4 eq. 5, and then multiplies it by a modulating factor (1 - p_t) raised to gamma eq. 6. The text sets alpha = 0.25 and gamma = 2 §4.1.

6. Optimisation. Adam with learning rate 0.001, trained for 2000 epochs in each run §4.1 §3.1.3.

# Pipeline per resampling method  [§3.1] [fig. 2]
# Input:  features X (N, d), labels y in {0, 1} (1 = minority)
# Output: mean test metrics over runs

def run_pipeline(X, y, resampler, runs=100):
    # Resample first, as drawn in the paper's pipeline  [fig. 2]
    X_res, y_res = resampler(X, y)
    # Scale each feature to [0, 1]; method not stated  [§3.1.2 Normalization]
    X_norm = scale_to_unit_interval(X_res)
    results = []
    for run in range(runs):
        # New random shuffle and split; ratio not stated  [§3.1.3]
        X_train, y_train, X_test, y_test = random_split(X_norm, y_res)
        # Either the DNN [tab. 2] or the CNN [tab. 3]
        model = make_model()
        optimizer = Adam(model.parameters(), lr=0.001)  # [§4.1]
        for epoch in range(2000):  # [§3.1.3]
            # Batch size and loss reduction are not stated  [§4.1]
            loss = focal_loss(model(X_train), y_train)
            optimizer.step(loss)
        # Eight metrics on the test split  [§4.4] [eq. 7] to [eq. 13]
        results.append(evaluate(model, X_test, y_test))
    return mean(results)  # average over runs  [§4.5]

# Focal loss  [eq. 4] [eq. 6]
# p: predicted probability of the minority class, shape (B,)
# y: labels in {0, 1}, shape (B,)

def focal_loss(p, y, alpha=0.25, gamma=2.0):
    # Probability of the true class  [eq. 4]
    p_t = where(y == 1, p, 1.0 - p)
    # alpha_t appears in [eq. 6] but its definition is not stated;
    # this card assumes a constant alpha for every sample
    alpha_t = alpha
    # Down-weight examples the model already gets right  [eq. 6]
    return -alpha_t * (1.0 - p_t) ** gamma * log(p_t)

# SMOTE  [§3.1.1 SMOTE] [fig. 4]
# X_min: minority samples, shape (M, d)
# n_new: number of synthetic samples needed for a 1:1 ratio

def smote(X_min, n_new, k=5):
    synthetic = []
    for _ in range(n_new):
        # Pick one minority sample at random  [§3.1.1 SMOTE, step 1]
        seed = random_choice(X_min)
        # Its k nearest minority neighbours, default k = 5  [§3.1.1 SMOTE, step 2]
        neighbours = k_nearest(seed, X_min, k)
        # Pick one neighbour at random  [§3.1.1 SMOTE, step 3]
        other = random_choice(neighbours)
        # Interpolation weight not stated in the paper; assumed uniform in [0, 1]
        weight = uniform(0.0, 1.0)
        synthetic.append(seed + weight * (other - seed))
    return synthetic

Hyperparameters

Datasets used in the paper tab. 4. The imbalance ratio is the majority count divided by the minority count, as the paper defines it §4.2.

Results

Author-reported. Setup: 24 imbalanced binary datasets from KEEL, a breast cancer set and Z-Alizadeh Sani tab. 4 §4.2; each value is the mean over 100 random shuffles and splits §4.5; training 2000 epochs with focal loss §4.1. The tables compare the six resampling pipelines with each other; no run without resampling is reported. Average rows are the paper's averages over datasets, as printed. Values are percentages. In the paper's tables the first value of each CNN/DNN pair is the CNN.

Bold marks the SMOTE CNN row, which the paper marks as its most robust model tab. 11. The ROS rows come from tab. 9 and are absent from tab. 11.

Per-dataset accuracy for the SMOTE pipeline, from tab. 5:

The per-dataset tables in the paper list 26 rows, including ecoli-0_vs_1 and vehicle0, which are not in tab. 4. The full per-dataset tables for every pipeline and metric are in tab. 5 to tab. 10.

Comparison on the Z-Alizadeh Sani dataset tab. 13, with the paper's own row for the SMOTE CNN pipeline: accuracy 98.57, recall 98.58, F1 98.57, precision 98.58, specificity 98.42 and AUC 99.14. The other rows are numbers reported by other studies; the paper does not restate their protocols, and it marks several entries as not considered. A separate comparison with the study by Mayabadi and Saadatfar (ref. 39) tab. 12 reports recall 99.00, G-mean 99.00 and F1 98.98 for the SMOTE CNN pipeline. The text ties that table to "16 imbalanced" datasets, but the paper does not list them.

The ROC plots fig. 18 are for the best-AUC run of each model; their AUC values range from 0.9105 for Nursery (CNN) to 1.0 for glass6 (CNN).

Paper vs code

No official code. The paper and the arXiv abstract page link no code or project page. A GitHub repository search on the title words, on "SMOTE CNN imbalance" and on the author name "joloudari" returned no matching repository, and the user search for the author name returned nothing. The card therefore reads only the paper.

Pitfalls

• Resampling before the split. The pipeline in fig. 2 and the text §3.1 apply resampling, then normalisation, then the random split. If that is what the code does, copies or synthetic points made from a test sample can sit in the training data, and test scores would be optimistic. The paper does not say that resampling is applied to the training part only. Implement the split first and resample only the training data unless the paper's order is confirmed.
• Normalisation is under-specified. The text says features are scaled to zero to one §3.1.2 Normalization, not which formula is used. It also does not say whether the scale is fitted on training data alone. A scale fitted on the full dataset would leak test information.
• No validation split and no stopping rule. The paper trains for 2000 epochs §3.1.3 but does not state a validation set, early stopping, or which epoch's weights are scored. The Adam settings beyond the learning rate are also not stated.
• Focal-loss alpha. eq. 6 writes alpha_t, but the paper never defines it. eq. 4 defines p_t and §4.1 gives a single alpha of 0.25. Whether alpha weights the minority class, the majority class, or both is not stated. This card's code assumes a constant alpha.
• Output activation. The paper writes the binary cross-entropy with predicted values in {0, 1} eq. 2, which describes labels, not probabilities. The focal loss needs a probability. Whether the output layer applies a sigmoid, or the loss is computed on logits, is not stated. Check which one is implemented before use.
• Wrong cross-reference. When rewriting the cross-entropy loss, the text points to "Equation (1)" §3.2.2. Equation 1 is the Tomek-link distance rule eq. 1. The intended reference is probably eq. 3.
• CNN dimensions do not line up. tab. 3 gives the second convolution 4 output channels but then a dense layer with 16 inputs. The flattened size after the convolutions, the padding, and the input layout of a tabular row (the length and channels given to a 1D convolution) are not stated. The architecture cannot be built exactly from the paper.
• ROS wording. §3.1.1 ROS says the class distributions are balanced by increasing the number of majority-class samples, but the rest of the paragraph describes copying minority samples. Read it as minority oversampling.
• Tomek links. The text says a link exists when "at least one" of the two inequalities in eq. 1 holds, and eq. 1 uses S_l, which is not defined in the text. The rule can therefore be read more loosely than a mutual-nearest-pair definition. Confirm the rule before implementing it.
• NearMiss variant. Three variants are described and NearMiss-2 is named as best in the cited study §3.1.2 NearMiss. Which variant produced tab. 8 is not stated.
• Dataset count. The text and tab. 4 list 24 datasets, while the per-dataset tables list 26 rows, including ecoli-0_vs_1 and vehicle0 tab. 5. The paper does not explain the difference, so the exact dataset set for the averages is uncertain.
• Averages hide spread. Each headline value is an unweighted mean over datasets. Nursery, at 88.71% accuracy for the best pipeline, pulls the mean down, and breast cancer, with 102,294 samples and an imbalance ratio of 163.2, counts the same as a 150-sample set tab. 4 tab. 5.
• Small gaps, no variance. The gap between SMOTE with the CNN (99.08) and the next pipeline (NearMiss with the DNN, 99.01) is 0.07 point tab. 11. The paper reports means only, with no standard deviation, confidence interval, or significance test over the 100 runs.
• Recall figure mismatch. tab. 13 gives recall 98.58 for the SMOTE CNN on Z-Alizadeh Sani. tab. 5 gives recall 98.57 for the same dataset and pipeline.
• External comparisons. tab. 12 and tab. 13 set the card's numbers next to other studies' reported numbers. The paper does not restate those studies' splits or preprocessing, so the comparisons are not shown to be like-for-like. The paper does not list the 16 datasets in tab. 12.
• Cost. The protocol is 2000 epochs for each of 100 runs, for each dataset and each of six resampling pipelines, for two models §3.1.3 §4.5. The paper reports the hardware in tab. 1 but not the total run time.

Lineage

• Builds on: the resampling methods (ROS, RUS, Tomek links, OSS, NearMiss and SMOTE), each cited to the original work in the paper's references §3.1. The focal loss is cited to its original paper as ref. 74 §3.2.2. The paper does not describe changes to these methods.
• Compared against: the other five resampling pipelines in the paper's own tables tab. 11. External comparisons are in tab. 12 and tab. 13.
• Superseded by: not stated.