Paper · 2022-04-07 · cs.LG
A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework
Survey and benchmark of 24 stream classifiers for class imbalance, run on 515 binary and multi-class streams with drift, dynamic imbalance and hard instances. Test-then-train over 200,000 instances, 500-instance windows, Kappa with G-Mean or PMAUC. Author-reported: ROSE is the most robust overall; boosting-based ensembles fare worst.
Concept driftData resampling for class imbalanceEnsemble learningdata-streamsclass-imbalanceconcept-driftbenchmark
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Some data never stops arriving. A stream like this is read once, in order, and the model has to learn as it goes. In many streams one class is rare, and the share of each class and the pattern behind the labels can both shift over time. Many methods have been proposed for this case, but they were usually tested on different setups, so it was hard to tell which ones to trust.
The authors built one shared test bench. It covers many artificial and real-world streams with rare classes, changing class shares, changing patterns and hard examples, and it reports several metrics that each show a different side of performance. They ran 24 published methods through it. No single method wins everywhere. One ensemble, ROSE, is the most consistently solid, and the boosting-based ensembles do the worst. The sections below give the protocol so you can rerun it, the numbers the authors report, and where the paper and its code disagree.
TL;DR
This is a survey and a benchmark for classifiers on imbalanced data streams. It defines 515 streams (binary and multi-class), a test-then-train protocol and the metrics Kappa, G-Mean and PMAUC, then compares 24 algorithms §1 tab. 2. Author-reported: ROSE is the most robust across scenarios §7.3; CSARF scores high on G-Mean (binary) and PMAUC (multi-class) but drops on Kappa §7.3. To compare a new method, use the same protocol and metrics, and read the per-scenario tables rather than the overall ranking. The authors' code is GPL-3.0; reimplement from this card, do not copy code.
Problem
Concept drift and class imbalance often arrive together in a stream: the classes can swap roles and the imbalance ratio can change over time, so methods that assume fixed data properties do not carry over §1. Existing studies usually cover few algorithms, mostly binary problems, and there is no agreed procedure or benchmark for evaluating them §1. Without one, a new method cannot be compared fairly with the state of the art §1. The protocol assumes each label is revealed right after its prediction, because every instance is used first to test and then to train §6.3.
Method
The paper has two parts. A taxonomy orders the field: streams by learning mode (chunk, online, hybrid) and by drift handling §2 tab. 2; imbalance approaches into data-level resampling and algorithm-level methods §4; and ensembles §5 fig. 3. An experimental framework then runs every algorithm on every stream family under one protocol §6. The steps below describe that framework, so an implementer can rebuild it.
- Streams. Ten generators are used: nine from MOA and one by Brzezinski. Each stream has 200,000 instances §6.2 tab. 4.
- Static imbalance. Each generator is run at imbalance ratios 1, 5, 10, 20, 50 and 100, with no drift §7.1.1 tab. 5.
- Dynamic imbalance. Four schedules of the ratio are run, each with gradual and sudden drift: increasing, increasing then decreasing, flipping, and flipping then reflipping. The exact sequences are in the Hyperparameters table §7.1.2 fig. 6 to fig. 9.
- Instance-level difficulty. The Brzezinski generator is run with rare minority instances, borderline minority instances, or both. Minority clusters are also moved, split or merged §7.1.3. The share of difficult instances is varied as listed in Hyperparameters.
- Drift with static imbalance §7.1.4 and drift with increasing imbalance §7.1.5: the same generators as in steps 2 and 3, combined.
- Real-world binary streams: 19 datasets tab. 11 §7.1.6. These differ from generated streams: their minority arrivals are not confined to bounded windows, and long single-class stretches can occur §7.1.6.
- Multi-class streams. Five classes, with Hyperplane, RandomRBF and RandomTree §7.2.1. The static distribution is listed in Hyperparameters. Shifting distributions rotate the same five-class mix §7.2.2. Drift is added to static and shifting imbalance §7.2.3 §7.2.4. The number of classes is varied §7.2.5. Real-world §7.2.6 tab. 18 and semi-synthetic §7.2.7 multi-class streams are also used.
- Algorithms. 24 binary-class algorithms, of which 15 handle multi-class §6.1 tab. 2. All use the settings their authors recommend, with no tuning §6.1. Ensembles use 10 base classifiers, each a Hoeffding tree §6.1.
- Evaluation. Test-then-train, instance by instance §6.3. The binary experiments report G-Mean and Kappa; the multi-class ones report PMAUC and Kappa. Each is computed over a sliding window of 500 instances §6.3. Accuracy, AUC, WMAUC and EWMAUC are on the authors' website only §6.3.
- Aggregation. Reported numbers are the median over five runs §6.1. Overall ranks combine the metrics with runtime and memory §7.3 fig. 45.
Metric definitions used in the study §6.3:
# Chance-corrected agreement over the window, scaled to 0-100 [§6.3] [eq. 1]
# Input: confusion (C, C); confusion[i, j] = count(true i, predicted j)
# Output: Kappa in [-100, 100]
def kappa_100(confusion):
# Number of examples in the window [eq. 1]
total = confusion.sum()
# Observed agreement: the diagonal count [eq. 1]
observed = confusion.trace()
# Agreement expected by chance from row and column totals [eq. 1]
row_totals = confusion.sum(axis=1)
col_totals = confusion.sum(axis=0)
chance = (row_totals * col_totals).sum()
return 100 * (total * observed - chance) / (total**2 - chance)# Multi-class AUC: mean pairwise AUC over ordered class pairs [§6.3] [eq. 5]
# Input: pair_auc[i][j] = AUC with class i positive, class j negative
# Output: PMAUC in [0, 1]
def pmauc(pair_auc, num_classes):
# Sum over all ordered pairs with i != j [eq. 5]
total = 0.0
for i in range(num_classes):
for j in range(num_classes):
if i != j:
total += pair_auc[i][j]
# Divide by the number of ordered pairs, C(C-1) [eq. 5]
return total / (num_classes * (num_classes - 1))Binary G-Mean is the square root of sensitivity times specificity, where sensitivity is TP/(TP+FN) and specificity is TN/(TN+FP) §6.3 eq. 4. Test-then-train loop as the paper describes it:
# Test-then-train on one stream [§6.3]
# Input: stream of (features, label) pairs, 200,000 long [§6.2]
# learner with predict_scores() and train_one()
# Output: window metrics as the stream runs
def run_stream(stream, learner):
# Keep the last 500 results for the metrics [§6.3]
recent = sliding_window(size=500)
for features, label in stream:
# Test first, with the model as it stands [§6.3]
scores = learner.predict_scores(features)
predicted = argmax(scores)
recent.add(label, predicted, scores)
# Then train on the same example [§6.3]
learner.train_one(features, label)
# Report the metrics over the window [§6.3]
yield recent.metrics()Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Stream length | 200,000 instances per stream | §6.2 tab. 4 | all generated streams |
| Evaluation protocol | test-then-train, one instance at a time | §6.3 | labels revealed after each prediction |
| Metric window | sliding window of 500 instances | §6.3 Static_Imbalance_Ratio.java | binary script checked |
| Metrics on the card | binary: G-Mean, Kappa; multi-class: PMAUC, Kappa | §6.3 | Accuracy, AUC, WMAUC, EWMAUC on the website only |
| Repetitions | median of 5 runs (5 seeds) | §6.1 | the repo script for static IR fixes seed 123456789 Static_Imbalance_Ratio.java |
| Ensemble base learner | Hoeffding tree | §6.1 | |
| Ensemble size | 10 base classifiers | §6.1 ROSE.java | default checked for ROSE only |
| Hyperparameter tuning | none; authors' recommended settings | §6.1 | |
| Static imbalance ratios | 1, 5, 10, 20, 50, 100 | §7.1.1 tab. 5 | see Pitfalls: stated ratios are approximate |
| Increasing ratio | 1, 5, 10, 20, 50, 100 | §7.1.2 | |
| Increasing then decreasing | 1, 5, 10, 20, 50, 100, 50, 20, 10, 5, 1 | §7.1.2 | |
| Flipping | 100, 50, 20, 10, 5, 1, 0.2, 0.1, 0.05, 0.02, 0.01 | §7.1.2 | values below 1 mean the roles have swapped |
| Flipping then reflipping | 100 down to 0.01, then back up to 100 | §7.1.2 | |
| Instance-level difficulty share | 0, 20, 40, 60, 80, 100 % (single); 0, 20, 40 % of each (combined) | §7.1.3 | |
| Multi-class static distribution | {50, 20, 10, 5, 1} over 5 classes | §7.2.1 | unit as written; the paper gives no further detail |
| Multi-class shifting | five rotations of the same distribution | §7.2.2 | sudden and gradual speeds |
| Number of classes | 3, 5, 10, 20, 30 | §7.2.5 | |
| Drift type | gradual and sudden | §7.1.2 §7.1.4 | drift position and width: not stated |
| Plot smoothing | moving average over 20 points | §7.1.2 | plots only, not the tables |
Results
Author-reported. Each table averages the streams of one experiment and lists the top algorithms the authors selected for it §7.1 §7.2. Ranks are the authors' averages over streams, and lower is better. Kappa and G-Mean are on the 0–100 scale. All algorithms ran with their recommended settings and no tuning §6.1.
Binary, static imbalance, average over all 10 streams tab. 5:
| Algorithm | Avg G-Mean | Avg Kappa | Rank G-Mean | Rank Kappa |
|---|---|---|---|---|
| CSARF | 91.01 | 57.85 | 3.08 | 6.40 |
| ARF | 63.53 | 58.52 | 8.08 | 6.49 |
| KUE | 69.85 | 64.55 | 7.87 | 6.00 |
| LB | 70.30 | 65.45 | 5.69 | 4.11 |
| CALMID | 70.50 | 64.53 | 6.32 | 4.77 |
| ROSE | 77.78 | 68.73 | 5.15 | 4.33 |
| ARFR | 69.66 | 56.15 | 5.93 | 6.32 |
| SMOTE-OB | 79.08 | 64.32 | 5.25 | 6.00 |
| OOB | 84.40 | 70.56 | 3.51 | 3.75 |
| UOB | 91.02 | 53.45 | 4.12 | 6.83 |
Binary, dynamic imbalance, average over all 10 streams tab. 6:
| Algorithm | Avg G-Mean | Avg Kappa | Rank G-Mean | Rank Kappa |
|---|---|---|---|---|
| CSARF | 87.02 | 50.65 | 3.83 | 8.58 |
| ARF | 72.02 | 63.86 | 8.62 | 6.31 |
| KUE | 78.02 | 67.53 | 7.19 | 5.14 |
| LB | 77.43 | 68.37 | 5.83 | 4.41 |
| CALMID | 75.96 | 67.59 | 6.85 | 4.68 |
| ROSE | 83.85 | 71.44 | 4.11 | 2.80 |
| ARFR | 75.00 | 62.59 | 6.66 | 6.41 |
| SMOTE-OB | 81.51 | 66.35 | 5.11 | 4.93 |
| OOB | 85.67 | 65.18 | 3.80 | 4.64 |
| UOB | 89.08 | 55.94 | 3.01 | 7.12 |
Binary, real-world datasets, average over 19 datasets tab. 12:
| Algorithm | Avg G-Mean | Avg Kappa | Rank G-Mean | Rank Kappa |
|---|---|---|---|---|
| CSARF | 61.41 | 37.76 | 1.50 | 4.08 |
| ARF | 44.01 | 35.53 | 6.95 | 4.84 |
| KUE | 38.09 | 27.72 | 9.39 | 8.87 |
| LB | 43.10 | 34.08 | 7.61 | 6.03 |
| CALMID | 44.04 | 33.73 | 7.61 | 6.34 |
| ROSE | 54.11 | 39.60 | 4.37 | 3.11 |
| ARFR | 53.38 | 35.36 | 5.13 | 5.39 |
| SMOTE-OB | 58.80 | 39.27 | 3.24 | 3.03 |
| OOB | 54.65 | 36.15 | 4.74 | 5.63 |
| UOB | 57.68 | 28.27 | 4.47 | 7.68 |
Multi-class, static imbalance (5 classes), average tab. 13:
| Algorithm | Avg PMAUC | Avg Kappa | Rank PMAUC | Rank Kappa |
|---|---|---|---|---|
| CSARF | 94.19 | 55.73 | 1.33 | 6.33 |
| ARF | 90.40 | 60.98 | 4.67 | 3.67 |
| KUE | 90.51 | 59.81 | 6.00 | 5.00 |
| LB | 85.89 | 50.06 | 8.33 | 7.67 |
| SRP | 89.55 | 53.86 | 5.33 | 7.00 |
| CALMID | 87.46 | 57.00 | 8.33 | 6.33 |
| MICFOAL | 87.75 | 58.14 | 8.00 | 4.67 |
| ROSE | 91.58 | 59.30 | 4.67 | 4.67 |
| ARFR | 93.13 | 63.47 | 2.67 | 3.00 |
| OOB | 91.75 | 56.64 | 5.67 | 6.67 |
Multi-class, real-world datasets, average tab. 19:
| Algorithm | Avg PMAUC | Avg Kappa | Rank PMAUC | Rank Kappa |
|---|---|---|---|---|
| CSARF | 43.65 | 30.02 | 4.06 | 7.00 |
| ARF | 42.93 | 32.72 | 5.22 | 5.39 |
| KUE | 41.39 | 28.68 | 6.89 | 6.83 |
| LB | 42.46 | 32.09 | 6.11 | 5.06 |
| SRP | 43.53 | 34.64 | 4.17 | 4.94 |
| CALMID | 42.17 | 42.03 | 5.56 | 4.11 |
| MICFOAL | 41.93 | 34.18 | 6.11 | 6.00 |
| ROSE | 42.04 | 28.67 | 6.17 | 5.17 |
| ARFR | 43.26 | 32.66 | 5.11 | 5.22 |
| OOB | 42.79 | 32.08 | 5.61 | 5.28 |
Runtime and memory, averaged over streams tab. 21. Runtime is in seconds per 1,000 instances; memory is in RAM-hours. Ranks are the authors' averages. Multi-class columns are blank where the algorithm is not multi-class.
| Algorithm | Bin. runtime (s) | Bin. memory (RAM-h) | Bin. runtime rank | Bin. memory rank | MC runtime (s) | MC memory (RAM-h) |
|---|---|---|---|---|---|---|
| IRL | 0.15 | 3.93E-05 | 5.34 | 8.13 | -- | -- |
| C-SMOTE | 18.01 | 2.39E-02 | 18.98 | 20.75 | -- | -- |
| VFC-SMOTE | 2.51 | 2.52E-03 | 14.48 | 18.97 | -- | -- |
| CSARF | 3.97 | 1.12E-02 | 14.56 | 18.83 | 3.35 | 5.94E-06 |
| GHVFDT | 0.01 | 1.16E-08 | 1.14 | 1.78 | 0.09 | 9.92E-11 |
| HDVFDT | 0.01 | 2.85E-08 | 2.14 | 3.21 | 0.08 | 2.14E-10 |
| ARF | 3.65 | 8.42E-03 | 14.43 | 18.65 | 3.14 | 1.31E-06 |
| KUE | 0.11 | 4.93E-06 | 7.34 | 7.79 | 0.28 | 1.64E-08 |
| LB | 0.13 | 7.30E-06 | 8.17 | 9.16 | 0.33 | 1.21E-08 |
| OBA | 0.05 | 2.60E-07 | 5.03 | 5.91 | 0.25 | 2.25E-08 |
| SRP | 3.51 | 6.27E-03 | 15.03 | 18.54 | 5.34 | 8.56E-06 |
| ESOS-ELM | 2.61 | 1.11E-06 | 15.29 | 9.41 | -- | -- |
| CALMID | 0.09 | 2.80E-06 | 7.13 | 8.21 | 0.25 | 2.25E-08 |
| MICFOAL | 1.91 | 3.37E-03 | 12.72 | 16.98 | 1.95 | 3.72E-06 |
| ROSE | 0.13 | 6.45E-06 | 8.70 | 10.03 | 0.49 | 2.82E-08 |
| OADA | 24.48 | 2.10E-02 | 20.45 | 15.23 | 97.24 | 1.54E-04 |
| OADAC2 | 31.61 | 2.11E-02 | 21.57 | 15.81 | -- | -- |
| ARFR | 1.25 | 8.73E-04 | 11.86 | 16.71 | 2.94 | 1.13E-06 |
| SMOTE-OB | 46.71 | 2.78E-01 | 21.10 | 21.08 | -- | -- |
| OSMOTE | 113.38 | 3.66E-01 | 23.29 | 15.70 | -- | -- |
| OOB | 0.07 | 1.94E-06 | 6.07 | 6.69 | 0.25 | 6.05E-08 |
| UOB | 0.03 | 3.55E-07 | 3.99 | 4.70 | 0.08 | 1.08E-09 |
| ORUB | 14.18 | 4.20E-03 | 18.57 | 12.32 | -- | -- |
| OUOB | 52.58 | 3.18E-01 | 22.60 | 15.41 | -- | -- |
Headline findings, all author-reported §7.3 fig. 45:
- ROSE gave the best single Kappa and was among the best on the other metrics, which the authors take as robustness across scenarios §7.3.
- CSARF was strong on G-Mean (binary) and PMAUC (multi-class), but its Kappa rank dropped sharply, and it becomes unsuitable at a high number of classes §7.3.
- OOB, UOB, ROSE and CALMID gave the best trade-off between accuracy and resource use §7.3. The lightest are the single-tree methods GHVFDT and HDVFDT §7.3 tab. 21.
- OADA was the weakest on the classification metrics, which the authors read as a limit of boosting-based ensembles §7.3.
- Oversampling methods that use SMOTE were the costliest in time and memory. SMOTE-OB, a strong method, used so much RAM that the authors call it prohibitive for many real-world uses §7.3.
- Multi-class: UOB was unusable on multi-class streams §7.2.1. SMOTE-OB does not support multi-class at all §7.3.
Paper vs code
The official repository is canoalberto/imbalanced-streams at the commit in
the frontmatter. It is licensed GPL-3.0, so it is copyleft; this card quotes no
code and cites by path and line only. The checks below are from reading
the code, not from running it.
- Ratio labels differ from the actual ratios. The paper defines the
imbalance ratio as the majority-to-minority size ratio §7.1.1. The
generator's
-ioption is the minority share, documented as "10%, i.e. IR 10" BinaryImbalancedGenerator.java, and each instance is a minority example with probability-iBinaryImbalancedGenerator.java. The script labels the 0.20 share "IR 5", the 0.10 share "IR 10", and so on Static_Imbalance_Ratio.java. Those shares give a majority-to-minority ratio of about 4, 9, 19, 49, 99, not 5, 10, 20, 50, 100. Implement from the share, not the label. - Kappa scale. Eq. 1 multiplies by 100 §6.3. The binary evaluator returns
the unscaled value
(p0 - pc) / (1 - pc)ImbalancedPerformanceEvaluator.java, and the multi-class evaluator does the same MultiClassImbalancedPerformanceEvaluator.java. The tables are on the 0–100 scale tab. 5. Where the factor 100 is applied in reporting was not checked. - Multi-class G-Mean root. The paper says the multi-class geometric mean is a square root §6.3. The code takes the C-th root of the product of class recalls MultiClassImbalancedPerformanceEvaluator.java. The paper reports PMAUC and Kappa, not G-Mean, for multi-class §6.3, so the tables are unaffected.
- PMAUC with a missing class. The code averages pairwise AUCs over the C(C-1) ordered pairs MultiClassImbalancedPerformanceEvaluator.java. A pair with no positive or no negative examples in the window scores 0 and still counts in the average MultiClassImbalancedPerformanceEvaluator.java. The paper does not mention this.
- Seeds. The static-IR script fixes one seed, 123456789 Static_Imbalance_Ratio.java. The README links result files for five seeds README.md, which matches the paper's median of five runs §6.1.
- Window and stream length match the paper: 500 and 200,000 Static_Imbalance_Ratio.java.
- Ensemble size. ROSE's default is 10 trees, matching §6.1 ROSE.java. The other ensembles were not checked.
- Label mismatch. Table 2 and the text call one method ORUS tab. 2; the
results table, the README and the code call it ORUB, after
OnlineRUSBoosttab. 21 Static_Imbalance_Ratio.java. They are the same method. - Not stated in the paper, not checked in the code: drift position and width, the segment lengths for dynamic schedules, and Hoeffding-tree settings.
Pitfalls
- Use the share, not the label. As noted above, the labelled ratios (5, 10, 20, 50, 100) are rounded from the minority shares. A replication that takes the labels literally gets slightly different streams.
- Kappa and G-Mean disagree under high imbalance. Kappa depends on the class distribution, which makes values hard to compare across imbalance levels. G-Mean and AUC do not reflect the imbalance level. Kappa penalises false positives more strongly at high imbalance, so a method can have high AUC and low Kappa §6.3. Report at least two complementary metrics.
- Read the per-scenario tables. The text says ROSE gives the best Kappa §7.3. That holds for the dynamic-imbalance table (71.44) tab. 6, but in the static table OOB has the highest average Kappa (70.56 against ROSE's 68.73) tab. 5.
- Overall ranks hide spread. The overall barplot and ranks combine four metrics with equal weight fig. 45. A method can rank well overall and still lose badly on one scenario, as CSARF does on Kappa.
- Real-world streams behave differently. Their minority arrivals are not confined to bounded windows, so generated-stream results may not transfer §7.1.6.
- Runtime and memory are hardware-specific. The authors ran on a cluster §6.1. Use the ranks, not the absolute seconds or RAM-hours. Memory is measured with a Java sizing agent Static_Imbalance_Ratio.java.
- No tuning, so the baselines may be weak. All methods use authors' defaults §6.1. Tuned baselines could rank differently.
- Plots are smoothed. The dynamic-imbalance figures use a 20-point moving average §7.1.2; the tables are not smoothed.
- Table 2 has an undefined code. The caption defines BL, CS, TM, RI, the blind resampling codes and H, but not RB, which appears for ESOS-ELM and OUOB tab. 2.
- Licence of the code. The paper is CC BY 4.0. The framework code is GPL-3.0. Using or redistributing that code brings GPL obligations, so reimplement from the paper rather than copying code.
Lineage
- Builds on: the MOA framework, which the authors used for all algorithms §6.1; the Brzezinski generator for instance-level difficulties §6.2; the prequential AUC of Brzezinski et al. (2017) and its multi-class extension by Wang et al. (2020) §6.3.
- Compared against: the 24 algorithms in tab. 2 and tab. 3.
- Earlier comparisons the paper contrasts itself with: Wang et al. (2018), Brzezinski et al. (2021), Bernardo et al. (2021), and Cano and Krawczyk (2021) §1.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “A survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework” by Gabriel Aguiar, Bartosz Krawczyk, Alberto Cano (arXiv:2204.03719), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/canoalberto/imbalanced-streams at commit 4365d4757da4 (GPL-3.0 (copyleft; no code is quoted here)). Any quoted code is shown under that license.