---
kind: concept
slug: concept-drift
title: Concept drift
summary: >-
  When a stream's input-label relation or class balance changes over time, a model trained on old data loses accuracy. Reactive detectors reset prototypes on alarm (RRSLVQ). A benchmark on imbalanced streams finds ensembles behave differently under drift. Results are author-reported, and the two papers use different protocols.
tags: [stream-learning, drift-detection, class-imbalance, benchmark, kolmogorov-smirnov-test, learning-vector-quantization]
papers: ["2007.05432", "2204.03719"]
related: [prototype-learning, continual-learning, data-resampling, ensemble-learning]
status: draft
updated_at: 2026-11-15
---

## Overview

A stream classifier learns from data that keeps arriving. If the distribution
of inputs and labels changes, a model fitted to older data loses accuracy
[arxiv:2007.05432 §3.2]. Two questions follow. The first is when the change
happened, which a drift detector answers. The second is what the model does
next, which an adaptation step answers. The RRSLVQ paper evaluates both parts
in a test-then-train setting: each sample is predicted before the model learns
from it [arxiv:2007.05432 §5.2].

Drift is not limited to the input distribution. In imbalanced streams the class
shares can change, and the minority and majority classes can swap roles
[arxiv:2204.03719 §1]. Methods that assume a fixed class balance do not carry
over to such streams [arxiv:2204.03719 §1].

The RRSLVQ paper also assumes that every real drift changes the input
distribution p(x). A change that only affects p(y|x) is outside the detector's
design [arxiv:2007.05432 §3.2].

## Variants

### Reactive detection with prototype reset (RRSLVQ)

A prototype classifier runs online. A sliding window of recent samples is kept
and split into a recent part R and a part W sampled from older data. A
Kolmogorov-Smirnov test runs on each feature. The detector fires when any
feature's distance passes a threshold. For equal window sizes that threshold
is the square root of minus ln(alpha) over r [arxiv:2007.05432 §4.1,
eq:kstest]. On a detection, every prototype is set to the mean of R, and the
prototype labels are kept [arxiv:2007.05432 §4.3]. The paper's experiments use
n = 300, r = 30 and alpha = 0.0001 [arxiv:2007.05432 §5.2]. Card:
[Reactive RSLVQ](/papers/2007.05432).

### Other detectors compared with Naive Bayes

The RRSLVQ paper compares its KSWIN detector with ADWIN, DDM and EDDM. Each
detector signals drift to a Naive Bayes classifier, which is reset on each
signal [arxiv:2007.05432 §5.2]. Overall mean accuracy over 12 streams is 0.7837
for KSWIN, 0.7120 for ADWIN, 0.6750 for DDM and 0.6323 for EDDM
[arxiv:2007.05432 tab. 3]. The confusion counts differ sharply. KSWIN records
20,659 correct drift flags and 51,040 false alarms. ADWIN records 71 and 173.
DDM records 0 and 12. EDDM records 33 and 55 [arxiv:2007.05432 tab. 2].

### Other drift-handling variants

The RRSLVQ paper reports that prototype insertion, or prototype replacement
triggered by ADWIN, did not keep the model stable during drift
[arxiv:2007.05432 §5.3]. The authors conclude that both the KSWIN detector and
the prototype adaptation matter for stability [arxiv:2007.05432 §5.3].

### Benchmarking under drift and imbalance

The imbalanced-streams survey builds a shared test bench rather than a
detector. It uses ten stream generators, each with 200,000 instances
[arxiv:2204.03719 §6.2]. Its drift settings include imbalance-ratio schedules
that increase, rise then fall, flip, or flip and flip back, each with gradual
and sudden drift [arxiv:2204.03719 §7.1.2]. It also adds drift to static and
increasing imbalance [arxiv:2204.03719 §7.1.4, §7.1.5], and it includes 19
real-world binary datasets [arxiv:2204.03719 §7.1.6]. Evaluation is
test-then-train, instance by instance, over a sliding window of 500 instances.
Binary streams report G-Mean and Kappa; multi-class streams report PMAUC and
Kappa [arxiv:2204.03719 §6.3]. Reported numbers are medians over five runs
[arxiv:2204.03719 §6.1]. Card:
[Imbalanced data streams survey](/papers/2204.03719).

### Ensembles on imbalanced streams with drift

On the survey's dynamic-imbalance table, ROSE has the best average Kappa, 71.44,
and a G-Mean of 83.85 [arxiv:2204.03719 tab. 6]. CSARF has the best G-Mean on
that table, 87.02, but its Kappa is 50.65 [arxiv:2204.03719 tab. 6]. The
authors call ROSE the most robust across scenarios, and they call the
boosting-based OADA the weakest on the classification metrics
[arxiv:2204.03719 §7.3]. Rankings depend on the scenario. On the static
table, OOB has a higher average Kappa than ROSE, 70.56 against 68.73
[arxiv:2204.03719 tab. 5].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Small, inspectable model on a limited device; classes may be missing or unbalanced in a batch | Reactive per-feature KS detection with prototype reset | Memory grows as d times (m + n) and prediction time with m [arxiv:2007.05432 §4.5]. The paper claims constant model size and time and tolerance of missing classes [arxiv:2007.05432 §4.3, §5.3]. |
| Highest accuracy on the RRSLVQ paper's streams; memory is not a constraint | Another stream classifier from the comparison, such as SAMKNN | SAMKNN has the best overall mean accuracy and kappa in that paper, 0.8581 and 0.6569. RRSLVQ scores 0.7455 and 0.4515 [arxiv:2007.05432 tab. 4, tab. 5]. |
| Only a drift signal is needed, and missed drifts cost more than false alarms | KSWIN-style per-feature test | Highest overall mean accuracy with Naive Bayes, and the most true detections [arxiv:2007.05432 tab. 2, tab. 3]. Expect 51,040 false alarms over the comparison [arxiv:2007.05432 tab. 2]. |
| False alarms are expensive | DDM or EDDM | Only 12 and 55 false alarms [arxiv:2007.05432 tab. 2]. DDM flags no true drift at all, and EDDM flags 33 [arxiv:2007.05432 tab. 2]. |
| Imbalanced stream with drift; one ensemble that performs consistently across scenarios | ROSE | Best average Kappa on the dynamic-imbalance table [arxiv:2204.03719 tab. 6], and the authors call it the most robust overall [arxiv:2204.03719 §7.3]. |
| Imbalanced stream with drift; G-Mean matters more than Kappa | CSARF, with care | Best G-Mean on the dynamic-imbalance table, but its Kappa rank drops sharply [arxiv:2204.03719 tab. 6, §7.3]. |
| Memory is tight on an imbalanced stream | Single-tree methods such as GHVFDT or HDVFDT; avoid SMOTE-based oversampling | The authors list the single-tree methods as the lightest [arxiv:2204.03719 §7.3, tab. 21]. The SMOTE-OB method used the most RAM, which the authors call prohibitive for many real-world uses [arxiv:2204.03719 §7.3]. |
| Comparing new stream methods under drift and imbalance | The survey's protocol: test-then-train, 500-instance window, Kappa with G-Mean or PMAUC | The protocol and metrics are defined so methods can be compared on the same footing [arxiv:2204.03719 §6.3]. |

The RRSLVQ numbers are single runs with no variance or confidence intervals
[arxiv:2007.05432 §5]. Differences of a point or two between methods should
not be treated as established.

## Implementation sketch

The core below follows the RRSLVQ paper's algorithm for one stream step
[arxiv:2007.05432 alg. 1]. Where the released code differs, see Caveats.

```pseudocode
# One stream step of reactive adaptation  [arxiv:2007.05432 alg. 1]
def process_sample(model, window, x, y):
    # Predict with the model as it is before this sample  [§5.2]
    y_hat = nearest_prototype_label(model, x)
    # Push the sample; the oldest one leaves a window of size n  [§4.1]
    window.push(x, y)
    # R: the r newest samples; W: r samples drawn from the older part  [§4.1]
    R, W = recent_and_sampled_older(window, r)
    # Fire if any feature's KS distance passes the threshold  [§4.1, eq:kstest]
    threshold = sqrt(-ln(alpha) / r)
    if any(ks_distance(W[:, i], R[:, i]) > threshold for i in range(d)):
        # Reset every prototype to the mean of R; keep the labels  [§4.3]
        model.theta[:] = R.mean(axis=0)
        # Train on each labelled sample in R (see Caveats)  [§4.3]
        for x_r, y_r in R:
            update_prototypes(model, x_r, y_r)
    # Ordinary RSLVQ update for the current sample  [§4.2]
    update_prototypes(model, x, y)
    return y_hat
```

The gradient and update rule are given in [arxiv:2007.05432 §4.2]. The
variants above differ in the detector and in what the reset step does.

Evaluation under drift uses the survey's test-then-train loop. This is our own
rendering of its protocol, not the authors' code [arxiv:2204.03719 §6.3].

```pseudocode
# Test-then-train with a sliding window  [arxiv:2204.03719 §6.3]
# Input:  stream of (x, y) pairs; learner with predict() and train()
# Output: window metrics after each instance

def evaluate(stream, learner, window_size=500):
    # Keep the last window_size outcomes for the metrics  [§6.3]
    recent = sliding_window(window_size)
    for x, y in stream:
        # Predict first, with the model as it stands  [§6.3]
        y_hat = learner.predict(x)
        recent.add(y, y_hat)
        # Then learn from the same labelled instance  [§6.3]
        learner.train(x, y)
        yield recent.kappa(), recent.g_mean()
```

## Caveats

- **Sampling limits the detector.** W is a uniform sample of the older part.
  A recurring stream can produce the same distribution in R and W, so a
  drift can go unseen. The authors note this and do not fix it
  [arxiv:2007.05432 §4.1].
- **Stability needs a lemma.** The stability argument assumes a linear
  relation between the two concepts' priors [arxiv:2007.05432 §4.4]. The
  paper does not test this assumption.
- **Bonferroni arithmetic does not hold.** The paper writes the corrected
  alpha as floor(alpha / r) = 0.05 / 60 = 0.0001. The division gives about
  0.00083, and the floor is 0 [arxiv:2007.05432 §4.1]. Use alpha directly, as
  the experiments do [arxiv:2007.05432 §5.2].
- **Released RRSLVQ code differs from the paper.** The code trains only on the
  current sample after a reset, not on R. Its kernel width and posterior
  normalisation also differ from the paper's formulas. Missing classes at
  start-up leave uninitialised prototypes. See
  [Reactive RSLVQ](/papers/2007.05432) for the full list.
- **Survey labels are rounded.** The survey labels its streams with imbalance
  ratios of 5, 10, 20, 50 and 100 [arxiv:2204.03719 §7.1.1]. The minority
  shares the generator uses give majority-to-minority ratios of about 4, 9, 19,
  49 and 99, as found by reading the released code. Implement from the share,
  not the label. See [Imbalanced data streams survey](/papers/2204.03719).
- **Multi-class evaluator scores empty pairs as zero.** The released code
  scores a class pair with no positive or no negative examples in the window as
  0 and still averages it in. The paper does not mention this. See the card's
  Paper vs code section.
- **Generated and real streams differ.** Results on generated streams may not
  transfer to real ones, whose minority arrivals are not confined to bounded
  windows [arxiv:2204.03719 §7.1.6].
- **Reimplement the survey, do not copy its code.** The framework code is
  GPL-3.0, so the protocol should be rebuilt from the paper
  [arxiv:2204.03719 §6].

## Contradictions

The two papers do not evaluate the same streams or the same protocol, so their
numbers are not directly comparable. The RRSLVQ paper runs one million samples
per stream and reports moving averages [arxiv:2007.05432 §5.3]. The survey
runs 200,000 instances per stream and reports windowed metrics over 500
instances [arxiv:2204.03719 §6.2, §6.3]. Both papers report ARF, but on
different streams, so its Kappa values cannot be compared across papers
[arxiv:2007.05432 tab. 5] [arxiv:2204.03719 tab. 5]. No direct disagreement
between the two is known.

Within the RRSLVQ paper, the text and the printed numbers do not always match.
The Bonferroni arithmetic above is one example [arxiv:2007.05432 §4.1]. Where a
claim about stability or accuracy depends on a single table entry, check the
table first.

Within the survey, the text says ROSE gives the best Kappa overall. That holds
on the dynamic-imbalance table [arxiv:2204.03719 tab. 6], but on the static
table OOB has the higher average Kappa [arxiv:2204.03719 tab. 5]. Read the
per-scenario tables rather than the overall ranking.

## Papers

- [Reactive RSLVQ](/papers/2007.05432): proposes RRSLVQ, a prototype classifier with a per-feature KS drift detector and prototype reset, and compares it with ADWIN, DDM, EDDM and several stream classifiers.
- [Imbalanced data streams survey](/papers/2204.03719): a survey and benchmark of 24 stream classifiers for class imbalance under drift, with a shared test-then-train protocol and Kappa, G-Mean and PMAUC metrics.
