Foundations of data imbalance and solutions for a data democracy

In plain words

Many real datasets have one outcome that is rare, such as fraud, a filed insurance claim, or a defect. A model can reach a high score just by always answering with the common outcome, and still find none of the rare cases. This chapter explains that trap and shows better ways to measure a model: counts of right and wrong answers per class, and scores such as precision and recall built from them.

It then tries common ways to fix the imbalance by changing the training data: removing common cases, copying rare cases, or inventing new rare cases from similar ones. The test uses an insurance-claims dataset. The fixes trade some headline accuracy for the ability to find rare cases, and some of the simpler removal methods do not find any. The chapter is a tutorial with Python snippets, not a new method.

TL;DR

Book chapter and tutorial, not a new method. It defines the imbalance ratio and the metrics for imbalanced data §2.1 §3. It compares five resampling methods against a no-resampling baseline with logistic regression on the Porto Seguro Safe Driver Prediction data §4 p. 6. Accuracy alone rewards the majority-only classifier. Several reported precision, recall and G-mean values do not match the chapter's own confusion matrices (see Pitfalls).

Problem

Classifiers assume that maximising accuracy is the right goal and that test data follows the training distribution. Both break when one class is rare: the fraud example is a rare positive class that a classifier can ignore §1. The chapter attributes the difficulty to four factors, of which it singles out two: the degree of class imbalance and the complexity of the concept §2. Complexity grows with class overlap and with small disjuncts, where the minority class splits into sub-concepts §2.2 fig. 1, p. 3. Accuracy hides the problem: with a rare positive class it can look high while nothing positive is found §3.1 tab. 2, p. 7.

Method

Imbalance measures. The imbalance ratio (IR) is the number of negative examples divided by the number of positive examples §2.1. The chapter writes it as

IR = \frac{\text{negative examples}}{\text{positive examples}} §2.1

Metrics. The chapter defines the binary confusion matrix (rows: actual, columns: predicted, with TN, FP, FN, TP) §3.1 tab. 1, p. 3. From it:

\text{Accuracy} = \frac{TN + TP}{TN + FP + FN + TP} §3.1

\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN} §3.2

F_\beta = (1 + \beta^2)\, \frac{\text{precision} \cdot \text{recall}} {\beta^2 \cdot \text{precision} + \text{recall}} §3.3

The chapter uses F1 (β = 1). Its G-measure is written as the geometric mean of precision and recall, \sqrt{\text{precision} \cdot \text{recall}} §3.3. The code computes G-mean with a library function instead (see Paper vs code). ROC curves plot the true positive rate against the false positive rate over all thresholds; AUC is the area under that curve, where 0.5 is random guessing §3.4 fig. 2, p. 5.

Undersampling removes majority-class rows §4.1.

• Random undersampling removes majority rows at random until the classes are equal §4.1.1.
• Tomek links remove the majority member of a pair of nearest opposite-class instances, where no third instance is closer to either one §4.1.2.
• Edited nearest neighbours (ENN) removes a majority instance when most of its k nearest neighbours are minority instances. The chapter's default k is 3 §4.1.3.

# Tomek links: drop the majority member of each link  [§4.1.2]
# Input:  X (n, d) features; y (n,) labels, 1 = minority, 0 = majority
# Output: X, y with majority members of Tomek links removed

def remove_tomek_links(X, y):
    # A pair of instances from different classes is a Tomek link when
    # no third instance is closer to either of them than they are to
    # each other  [§4.1.2]
    for each pair (a, b) with y[a] != y[b]:
        if is_tomek_link(X, a, b):
            # Remove only the majority member  [§4.1.2]
            drop the instance of the pair with label 0
    return X, y

# Edited nearest neighbours  [§4.1.3]
# Input:  X (n, d); y (n,) with 1 = minority, 0 = majority
# Output: X, y with some majority instances removed

def edited_nearest_neighbours(X, y, n_neighbors=3):
    for each majority instance i:
        # Find its closest n_neighbors instances  [§4.1.3]
        neighbours = k_nearest(X, i, n_neighbors)
        # Remove i when most neighbours are minority  [§4.1.3]
        # (strict majority; the chapter does not say how ties are handled)
        if sum(y[neighbours]) > n_neighbors / 2:
            mark i for removal
    return the unmarked rows of X and y

The chapter says "most" without a tie rule; for an even k, a tie is not decided §4.1.3. The pseudocode takes a strict majority, which is an assumption of this card.

Oversampling adds minority-class rows §4.2.

• Random oversampling copies minority rows chosen at random until the classes are equal §4.2.1.
• SMOTE creates new minority points by interpolating between a minority instance and nearby minority instances in feature space. The chapter defers the interpolation details to the original SMOTE paper, which it cites; they are not stated here §4.2.2.
• ADASYN is an extension of SMOTE that generates more synthetic points for minority instances that are harder to learn, judged by their density distribution §4.2.2 §4.2.3.

Hybrid methods combine the two families. The chapter names SMOTE+ENN and SMOTE+Tomek and points to the library for them; it does not evaluate them §4.3.

Algorithm-level and ensemble approaches (cost-sensitive learning, one-class learning, bagging, boosting) are only named, not covered §5.

Reported experiment. The chapter fits logistic regression on the Porto Seguro Safe Driver Prediction data (about 595k rows; target 1 for about 21.7k rows and 0 for about 573.5k) after selecting 22 of 59 columns §3 p. 6. It then applies each method and reports accuracy, precision, recall, F1, G-mean and AUC §4 p. 6.

# Experiment as the chapter reports it  [§4] [p. 6] [tab. 4]
# Input:  features X (n, d); labels y with 1 = positive (minority)
# Output: test metrics for logistic regression

def resampling_experiment(X, y, sampler):
    # Resample the full dataset before splitting. The table totals
    # imply this order; see Pitfalls  [tab. 4] [p. 8]
    X_res, y_res = sampler.fit_resample(X, y)

    # Hold out 30 percent for testing, seed 0  [p. 6]
    X_train, X_test, y_train, y_test = split_30_percent(X_res, y_res)

    # Logistic regression with the listed settings  [p. 6]
    model = LogisticRegression(C=1.0, penalty="l2", solver="liblinear")
    model.fit(X_train, y_train)

    # Hard labels for accuracy, precision, recall, F1, G-mean  [p. 6]
    y_pred = model.predict(X_test)

    # Probabilities for the ROC curve and AUC  [p. 7]
    scores = model.predict_proba(X_test)[:, 1]
    return metrics(y_test, y_pred, scores)

For the correct procedure, split first and resample only the training portion. The chapter does not state that it did this, and the table totals suggest it did not (see Pitfalls).

Hyperparameters

Results

Author-reported. Setup: Porto Seguro Safe Driver Prediction; logistic regression; 70/30 split; one run per method; baseline is logistic regression without resampling. Values are as printed in the chapter, before any correction noted in Pitfalls.

The chapter's stated conclusion is that oversampling methods perform better than undersampling here, and that only random undersampling finds positives among the undersamplers §5, Conclusion p. 17–18. It also notes that no single method suits every problem p. 18.

The hybrid methods are not evaluated.

Paper vs code

No official code. The paper names no code repository. Its links are to the Kaggle dataset page, the imbalanced-learn documentation and a Python reference [refs. 16, 20, 21]. GitHub searches for "data democracy imbalance" and for the title keywords found no repository. The chapter's Python appears only as screenshots in the PDF (pp. 6, 8–16). The discrepancies below are between the chapter's text and its own screenshots.

• G-mean. The text defines G as the square root of precision times recall §3.3. The code calls geometric_mean_score from the imbalanced-learn library p. 6. The G-mean reported for random undersampling tab. 5 matches the square root of recall times specificity, computed from tab. 4, not the text's formula (see Pitfalls).
• Argument order of precision and recall. The code calls precision_score(y_pred, y_test) and recall_score(y_pred, y_test) p. 6. scikit-learn expects the true labels first. The reported precision and recall values are swapped in most tables (see Pitfalls).
• AUC. The plotted AUC comes from the ROC curve built on predicted probabilities p. 7. The code also calls roc_auc_score on the hard labels y_pred p. 7. The printed AUC is the curve value; the hard-label value is not reported.
• ADASYN class counts. The text says the positive class has 573,518 examples and the negative class 578,497 p. 16. The output in the code shows class 0 (negative) at 573,518 and class 1 (positive) at 578,497 p. 16.

Pitfalls

• Resampling before the split (inferred from counts). The resampling code calls fit_resample on the full data p. 8, p. 10, p. 13, p. 14, p. 16. The test totals are about 30 percent of the resampled sizes: random undersampling test total 13,017 in tab. 4 against 43,388 resampled rows p. 8; random oversampling 344,111 in tab. 8 against 1,147,036 p. 13. The baseline total 178,564 in tab. 2 is about 30 percent of the roughly 595k unresampled rows §3. This suggests the resampled test sets contain copies or synthetic points of training rows, which would inflate the resampling results. This is an inference from the totals; the chapter does not state the order. Implement the split first, resample only the training part, and re-run before comparing with these numbers.
• Precision and recall swapped relative to the chapter's own matrices. The definitions in §3.2 applied to tab. 4 (TN 4115, FP 2402, FN 2878, TP 3622) give precision 0.60 and recall 0.56. The table reports precision 0.56 and recall 0.60 tab. 5. The same pattern holds in tab. 9, tab. 11 and tab. 13 (approximate for tab. 13). The likely cause is the argument order in the code (see Paper vs code). F1 is symmetric and unaffected.
• F1 and G-mean in tab. 9 do not match its matrix. From the matrix (TN 108341, FP 63542, FN 77842, TP 94386), the F1 is about 0.57 and the G-mean about 0.59. The table reports F1 0.59 and G-mean 0.57. Check the values before using them.
• G-mean formula. The text's formula (square root of precision times recall) gives 0.58 for tab. 4, while the table reports 0.59. The square root of recall times specificity gives 0.59. Implement G-mean with the definition you intend, and say which one in your results.
• Zero scores are not a finding about precision. The baseline, Tomek links and ENN predict only the negative class (tab. 2, tab. 6: FP = 0, TP = 0). Accuracy of 0.96 is the majority rate. Precision is undefined when nothing is predicted positive. The chapter reports 0 and does not say how it handled that case.
• Single run, no variance. Each method is run once with a fixed seed, and no repeated splits or intervals are reported. AUC differences of 0.01 to 0.02 (0.62 to 0.64) and accuracy differences of 0.01 are not tested. The conclusion that oversampling is better rests on these single values.
• Feature selection is not described. The chapter drops 59 columns to 22 but states no criterion §3 p. 6. Results depend on this choice.
• Hybrid methods are not evaluated. SMOTE+ENN and SMOTE+Tomek are named only §4.3.
• Figure and table labels are reused or wrong. Figure 8 is used for two ROC plots (random undersampling on p. 9, ENN on p. 12). Figure 10 is used for the Tomek ROC (p. 11) and the ENN code (p. 11). Tables 6 and 7 are used for both Tomek (p. 10) and ENN (p. 12). The caption of Figure 14 says "Random Oversampling" but the plot is in the SMOTE section (p. 15). The captions of Figures 3 to 5 do not match the code they show (p. 6). Cite by page as well as number.
• Equation numbers do not exist. The formulas are unnumbered; cite the section §3.1 to §3.3.
• Interpolation details of SMOTE and ADASYN are not stated in this chapter; they are in the cited original papers.
• Attribution note. The PDF lists authors in the order Kulkarni, Batarseh, Chong; the arXiv metadata lists Kulkarni, Chong, Batarseh. The front matter uses the metadata order.

Lineage

• Builds on: Japkowicz and Stephen's study of the four factors behind imbalance problems, cited as ref. 14 §2. The undersampling, oversampling and hybrid methods are named and cited from Fernández et al. (ref. 5), Tomek (ref. 22), Wilson (ref. 23), Chawla et al. (SMOTE, ref. 24) and He et al. (ADASYN, ref. 26), as described in this chapter only.
• Compared against: the no-resampling logistic regression baseline inside this chapter. No other method comparison.
• Superseded by: not stated in the chapter.