wikillm

Concept · 2 papers · updated 2026-11-15

Concept drift

When a stream's input-label relation or class balance changes over time, a model trained on old data loses accuracy. Reactive detectors reset prototypes on alarm (RRSLVQ). A benchmark on imbalanced streams finds ensembles behave differently under drift. Results are author-reported, and the two papers use different protocols.

stream-learningdrift-detectionclass-imbalancebenchmarkkolmogorov-smirnov-testlearning-vector-quantization

Markdown8 min read

AI-compiled overview, updated 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

A stream classifier learns from data that keeps arriving. If the distribution of inputs and labels changes, a model fitted to older data loses accuracy 2007.05432 §3.2. Two questions follow. The first is when the change happened, which a drift detector answers. The second is what the model does next, which an adaptation step answers. The RRSLVQ paper evaluates both parts in a test-then-train setting: each sample is predicted before the model learns from it 2007.05432 §5.2.

Drift is not limited to the input distribution. In imbalanced streams the class shares can change, and the minority and majority classes can swap roles 2204.03719 §1. Methods that assume a fixed class balance do not carry over to such streams 2204.03719 §1.

The RRSLVQ paper also assumes that every real drift changes the input distribution p(x). A change that only affects p(y|x) is outside the detector's design 2007.05432 §3.2.

Variants

Reactive detection with prototype reset (RRSLVQ)

A prototype classifier runs online. A sliding window of recent samples is kept and split into a recent part R and a part W sampled from older data. A Kolmogorov-Smirnov test runs on each feature. The detector fires when any feature's distance passes a threshold. For equal window sizes that threshold is the square root of minus ln(alpha) over r [arxiv:2007.05432 §4.1, eq:kstest]. On a detection, every prototype is set to the mean of R, and the prototype labels are kept 2007.05432 §4.3. The paper's experiments use n = 300, r = 30 and alpha = 0.0001 2007.05432 §5.2. Card: Reactive RSLVQ.

Other detectors compared with Naive Bayes

The RRSLVQ paper compares its KSWIN detector with ADWIN, DDM and EDDM. Each detector signals drift to a Naive Bayes classifier, which is reset on each signal 2007.05432 §5.2. Overall mean accuracy over 12 streams is 0.7837 for KSWIN, 0.7120 for ADWIN, 0.6750 for DDM and 0.6323 for EDDM 2007.05432 tab. 3. The confusion counts differ sharply. KSWIN records 20,659 correct drift flags and 51,040 false alarms. ADWIN records 71 and 173. DDM records 0 and 12. EDDM records 33 and 55 2007.05432 tab. 2.

Other drift-handling variants

The RRSLVQ paper reports that prototype insertion, or prototype replacement triggered by ADWIN, did not keep the model stable during drift 2007.05432 §5.3. The authors conclude that both the KSWIN detector and the prototype adaptation matter for stability 2007.05432 §5.3.

Benchmarking under drift and imbalance

The imbalanced-streams survey builds a shared test bench rather than a detector. It uses ten stream generators, each with 200,000 instances 2204.03719 §6.2. Its drift settings include imbalance-ratio schedules that increase, rise then fall, flip, or flip and flip back, each with gradual and sudden drift 2204.03719 §7.1.2. It also adds drift to static and increasing imbalance 2204.03719 §7.1.4, §7.1.5, and it includes 19 real-world binary datasets 2204.03719 §7.1.6. Evaluation is test-then-train, instance by instance, over a sliding window of 500 instances. Binary streams report G-Mean and Kappa; multi-class streams report PMAUC and Kappa 2204.03719 §6.3. Reported numbers are medians over five runs 2204.03719 §6.1. Card: Imbalanced data streams survey.

Ensembles on imbalanced streams with drift

On the survey's dynamic-imbalance table, ROSE has the best average Kappa, 71.44, and a G-Mean of 83.85 2204.03719 tab. 6. CSARF has the best G-Mean on that table, 87.02, but its Kappa is 50.65 2204.03719 tab. 6. The authors call ROSE the most robust across scenarios, and they call the boosting-based OADA the weakest on the classification metrics 2204.03719 §7.3. Rankings depend on the scenario. On the static table, OOB has a higher average Kappa than ROSE, 70.56 against 68.73 2204.03719 tab. 5.

When to use which

SituationUseWhy
Small, inspectable model on a limited device; classes may be missing or unbalanced in a batchReactive per-feature KS detection with prototype resetMemory grows as d times (m + n) and prediction time with m 2007.05432 §4.5. The paper claims constant model size and time and tolerance of missing classes 2007.05432 §4.3, §5.3.
Highest accuracy on the RRSLVQ paper's streams; memory is not a constraintAnother stream classifier from the comparison, such as SAMKNNSAMKNN has the best overall mean accuracy and kappa in that paper, 0.8581 and 0.6569. RRSLVQ scores 0.7455 and 0.4515 2007.05432 tab. 4, tab. 5.
Only a drift signal is needed, and missed drifts cost more than false alarmsKSWIN-style per-feature testHighest overall mean accuracy with Naive Bayes, and the most true detections 2007.05432 tab. 2, tab. 3. Expect 51,040 false alarms over the comparison 2007.05432 tab. 2.
False alarms are expensiveDDM or EDDMOnly 12 and 55 false alarms 2007.05432 tab. 2. DDM flags no true drift at all, and EDDM flags 33 2007.05432 tab. 2.
Imbalanced stream with drift; one ensemble that performs consistently across scenariosROSEBest average Kappa on the dynamic-imbalance table 2204.03719 tab. 6, and the authors call it the most robust overall 2204.03719 §7.3.
Imbalanced stream with drift; G-Mean matters more than KappaCSARF, with careBest G-Mean on the dynamic-imbalance table, but its Kappa rank drops sharply 2204.03719 tab. 6, §7.3.
Memory is tight on an imbalanced streamSingle-tree methods such as GHVFDT or HDVFDT; avoid SMOTE-based oversamplingThe authors list the single-tree methods as the lightest 2204.03719 §7.3, tab. 21. The SMOTE-OB method used the most RAM, which the authors call prohibitive for many real-world uses 2204.03719 §7.3.
Comparing new stream methods under drift and imbalanceThe survey's protocol: test-then-train, 500-instance window, Kappa with G-Mean or PMAUCThe protocol and metrics are defined so methods can be compared on the same footing 2204.03719 §6.3.

The RRSLVQ numbers are single runs with no variance or confidence intervals 2007.05432 §5. Differences of a point or two between methods should not be treated as established.

Implementation sketch

The core below follows the RRSLVQ paper's algorithm for one stream step 2007.05432 alg. 1. Where the released code differs, see Caveats.

Pseudocode
# One stream step of reactive adaptation  [arxiv:2007.05432 alg. 1]
def process_sample(model, window, x, y):
    # Predict with the model as it is before this sample  [§5.2]
    y_hat = nearest_prototype_label(model, x)
    # Push the sample; the oldest one leaves a window of size n  [§4.1]
    window.push(x, y)
    # R: the r newest samples; W: r samples drawn from the older part  [§4.1]
    R, W = recent_and_sampled_older(window, r)
    # Fire if any feature's KS distance passes the threshold  [§4.1, eq:kstest]
    threshold = sqrt(-ln(alpha) / r)
    if any(ks_distance(W[:, i], R[:, i]) > threshold for i in range(d)):
        # Reset every prototype to the mean of R; keep the labels  [§4.3]
        model.theta[:] = R.mean(axis=0)
        # Train on each labelled sample in R (see Caveats)  [§4.3]
        for x_r, y_r in R:
            update_prototypes(model, x_r, y_r)
    # Ordinary RSLVQ update for the current sample  [§4.2]
    update_prototypes(model, x, y)
    return y_hat

The gradient and update rule are given in 2007.05432 §4.2. The variants above differ in the detector and in what the reset step does.

Evaluation under drift uses the survey's test-then-train loop. This is our own rendering of its protocol, not the authors' code 2204.03719 §6.3.

Pseudocode
# Test-then-train with a sliding window  [arxiv:2204.03719 §6.3]
# Input:  stream of (x, y) pairs; learner with predict() and train()
# Output: window metrics after each instance

def evaluate(stream, learner, window_size=500):
    # Keep the last window_size outcomes for the metrics  [§6.3]
    recent = sliding_window(window_size)
    for x, y in stream:
        # Predict first, with the model as it stands  [§6.3]
        y_hat = learner.predict(x)
        recent.add(y, y_hat)
        # Then learn from the same labelled instance  [§6.3]
        learner.train(x, y)
        yield recent.kappa(), recent.g_mean()

Caveats

  • Sampling limits the detector. W is a uniform sample of the older part. A recurring stream can produce the same distribution in R and W, so a drift can go unseen. The authors note this and do not fix it 2007.05432 §4.1.
  • Stability needs a lemma. The stability argument assumes a linear relation between the two concepts' priors 2007.05432 §4.4. The paper does not test this assumption.
  • Bonferroni arithmetic does not hold. The paper writes the corrected alpha as floor(alpha / r) = 0.05 / 60 = 0.0001. The division gives about 0.00083, and the floor is 0 2007.05432 §4.1. Use alpha directly, as the experiments do 2007.05432 §5.2.
  • Released RRSLVQ code differs from the paper. The code trains only on the current sample after a reset, not on R. Its kernel width and posterior normalisation also differ from the paper's formulas. Missing classes at start-up leave uninitialised prototypes. See Reactive RSLVQ for the full list.
  • Survey labels are rounded. The survey labels its streams with imbalance ratios of 5, 10, 20, 50 and 100 2204.03719 §7.1.1. The minority shares the generator uses give majority-to-minority ratios of about 4, 9, 19, 49 and 99, as found by reading the released code. Implement from the share, not the label. See Imbalanced data streams survey.
  • Multi-class evaluator scores empty pairs as zero. The released code scores a class pair with no positive or no negative examples in the window as 0 and still averages it in. The paper does not mention this. See the card's Paper vs code section.
  • Generated and real streams differ. Results on generated streams may not transfer to real ones, whose minority arrivals are not confined to bounded windows 2204.03719 §7.1.6.
  • Reimplement the survey, do not copy its code. The framework code is GPL-3.0, so the protocol should be rebuilt from the paper 2204.03719 §6.

Contradictions

The two papers do not evaluate the same streams or the same protocol, so their numbers are not directly comparable. The RRSLVQ paper runs one million samples per stream and reports moving averages 2007.05432 §5.3. The survey runs 200,000 instances per stream and reports windowed metrics over 500 instances 2204.03719 §6.2, §6.3. Both papers report ARF, but on different streams, so its Kappa values cannot be compared across papers 2007.05432 tab. 5 2204.03719 tab. 5. No direct disagreement between the two is known.

Within the RRSLVQ paper, the text and the printed numbers do not always match. The Bonferroni arithmetic above is one example 2007.05432 §4.1. Where a claim about stability or accuracy depends on a single table entry, check the table first.

Within the survey, the text says ROSE gives the best Kappa overall. That holds on the dynamic-imbalance table 2204.03719 tab. 6, but on the static table OOB has the higher average Kappa 2204.03719 tab. 5. Read the per-scenario tables rather than the overall ranking.

Papers

  • Reactive RSLVQ: proposes RRSLVQ, a prototype classifier with a per-feature KS drift detector and prototype reset, and compares it with ADWIN, DDM, EDDM and several stream classifiers.
  • Imbalanced data streams survey: a survey and benchmark of 24 stream classifiers for class imbalance under drift, with a shared test-then-train protocol and Kappa, G-Mean and PMAUC metrics.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.