Data resampling for class imbalance

Overview

When one class is rare, a model can score high accuracy by predicting the common class every time. On the Porto Seguro Safe Driver data, the no-resampling logistic regression reaches accuracy 0.96 with no positive predictions 2108.00071 tab. 2. The imbalance ratio is the count of negatives divided by the count of positives 2108.00071 §2.1, and the chapter argues that accuracy, precision and recall should be read from the confusion matrix 2108.00071 §3.1 2108.00071 §3.2.

Resampling is the family of fixes that change the training data rather than the model. Undersampling removes majority rows, oversampling copies or creates minority rows, and hybrids combine the two 2108.00071 §4. Each variant below changes the class counts. The chapter does not state whether it resampled before or after the train/test split 2108.00071 §4, so the sketch in this page splits first. The SECOM card also oversamples before its feature selection step and does not say where the train/test split falls 2008.12987 §3.

Variants

Random undersampling

Removes majority rows at random until the classes are equal 2108.00071 §4.1.1. In the chapter's single run it was the only undersampler that found positives: accuracy 0.59 and AUC 0.63 against 0.96 and 0.62 without resampling 2108.00071 tab. 4 2108.00071 tab. 5.

Tomek links

Finds pairs of opposite-class instances where no third instance is closer to either member, and removes the majority member of each pair 2108.00071 §4.1.2. In the chapter it removed rows but left the classifier predicting only the majority class: accuracy 0.96, precision and recall 0, AUC 0.62 2108.00071 tab. 6 2108.00071 tab. 7.

Edited nearest neighbours (ENN)

Removes a majority instance when most of its k nearest neighbours are minority instances. The chapter's default is k = 3, and it does not say how a tie is handled for even k 2108.00071 §4.1.3. As with Tomek links, the classifier still predicted only the majority class: accuracy 0.96, AUC 0.64 2108.00071 tab. 6 2108.00071 fig. 8, p. 12.

Random oversampling

Copies minority rows at random until the classes are equal 2108.00071 §4.2.1. Accuracy 0.59, precision 0.54 and recall 0.60 as printed, AUC 0.63 2108.00071 tab. 8 2108.00071 tab. 9. The chapter's conclusion is that oversampling did better than undersampling here 2108.00071 §5. The gap is 0.01 AUC, from one run each.

SMOTE

Creates new minority points by interpolating between a minority instance and nearby minority instances. The chapter defers the interpolation details to the original SMOTE paper and does not state them 2108.00071 §4.2.2. Accuracy 0.58, AUC 0.62 2108.00071 tab. 10 2108.00071 tab. 11. See the SMOTE concept page for the one other card that uses it.

ADASYN

An extension of SMOTE that makes more synthetic points for minority instances judged harder to learn, using their density distribution 2108.00071 §4.2.2 2108.00071 §4.2.3. Accuracy 0.58, AUC 0.62 2108.00071 tab. 12 2108.00071 tab. 13.

Density-based SMOTE (DBSMOTE)

A density-based SMOTE variant. The SECOM paper uses it to oversample the failure class before a genetic-algorithm feature selection, on 104 failure cases 2008.12987 §3.2. Its neighbour count and sampling amount are not stated 2008.12987 §3.2. The paper does not run it with and without oversampling, so it gives no measured effect of this step. Card: AI-based Modeling and Data-driven Evaluation for Smart Manufacturing Processes. See the SMOTE concept page for the base method.

Hybrid methods and named-only approaches

SMOTE+ENN and SMOTE+Tomek are named and left to a library, and are not evaluated 2108.00071 §4.3. Cost-sensitive learning, one-class learning, bagging and boosting are named but not covered 2108.00071 §5. For loss-side weighting, see class-balanced loss.

Reported results

Author-reported, one run per method, logistic regression, 70/30 split, seed 0. The values are as printed, before the corrections under Contradictions. All rows are from 2108.00071.

The differences among the resampled methods are 0.01 to 0.02 AUC and 0.01 accuracy. None of these were tested for significance, and no intervals are reported. The hybrids are not in the table because they were not evaluated.

The SECOM card is not in this table. Its tables compare feature-selection methods, not resampling methods 2008.12987 tab. 2. Its 90.2% test accuracy comes from the full pipeline with DBSMOTE as one stage, and it is not comparable with the Porto Seguro numbers above 2008.12987 tab. 2.

When to use which

Implementation sketch

The chapter's experiment uses a 30 percent test split with seed 0 and logistic regression with C = 1.0 2108.00071 p. 6. The split-first order below is this page's design choice, because the chapter does not state its order 2108.00071 §4.

# Split first, then resample only the training part
# Split settings as in the chapter  [arxiv:2108.00071 p. 6]
def fit_with_resampling(X, y, sampler):
    X_tr, X_te, y_tr, y_te = split(X, y, test_size=0.3, seed=0)

    # Undersampler [arxiv:2108.00071 §4.1] or oversampler [§4.2]
    # SMOTE and ADASYN interpolation details: not stated here [§4.2.2]
    X_res, y_res = sampler.fit_resample(X_tr, y_tr)

    # Logistic regression settings as printed  [arxiv:2108.00071 p. 6]
    model = LogisticRegression(C=1.0, penalty="l2", solver="liblinear")
    model.fit(X_res, y_res)

    # Report counts and derived scores, not accuracy alone  [§3.1] [§3.2]
    return confusion_matrix(y_te, model.predict(X_te))

Contradictions

• Precision and recall appear swapped. Applying the definitions in 2108.00071 §3.2 to the matrix in 2108.00071 tab. 4 (TN 4115, FP 2402, FN 2878, TP 3622) gives precision 0.60 and recall 0.56. The table prints precision 0.56 and recall 0.60 2108.00071 tab. 5. The same pattern appears in 2108.00071 tab. 9, and the chapter's code appears to pass its arguments in reverse order 2108.00071 p. 6. The table values are shown above as printed.
• G-mean has two definitions that disagree with the table. The text defines G as the square root of precision times recall, which gives 0.58 for 2108.00071 tab. 4. The table prints 0.59, which matches the square root of recall times specificity 2108.00071 §3.3 2108.00071 tab. 5.
• F1 and G-mean in tab. 9 do not match its matrix. The matrix gives an F1 of about 0.57 and a G-mean of about 0.59, while the table prints 0.59 and 0.57 2108.00071 tab. 9.
• Possible resampling before the split. The test totals in 2108.00071 tab. 4 and 2108.00071 tab. 8 are about 30 percent of the resampled row counts in the text 2108.00071 p. 8 2108.00071 p. 13. That suggests the test sets may include copies or synthetic points of training rows. This is an inference from the totals, not something the chapter states, and it would make those scores not comparable with a split-first run.
• Zero precision is reported, not explained. Where a method predicts no positives, precision is undefined. The chapter reports 0 and does not say how it handled that case 2108.00071 tab. 2 2108.00071 tab. 6.

Papers

• Foundations of data imbalance and solutions for a data democracy: a tutorial chapter that defines the imbalance ratio and metrics, then compares five resampling methods with a no-resampling baseline on Porto Seguro data; single run, author-reported.
• AI-based Modeling and Data-driven Evaluation for Smart Manufacturing Processes: a GA feature-selection pipeline on SECOM sensor data, where density-based SMOTE oversamples the failure class; author-reported, with no resampling-only comparison.