Data stream processing has become central to modern machine learning applications, with concept drifts and novel class appearances posing the primary challenges faced by recognition methods. Adapting to concept drifts while also being able to detect novel classes and distinguish groups of unknown observations can be seen as a step towards effective method employment in an open-world setting. This work proposes a classification method that can detect and learn the distributions of new, unknown classes, as well as adapt to concept drifts once the distribution of known classes shifts. The proposed method uses the autoencoders' reconstruction errors and the KDE that models their distribution. The autoencoders incrementally adapt to the changing distributions of known classes via weight fine-tuning, resulting in continuous adjustment to evolving concepts. Conducted experiments used a diverse set of synthetic data streams, embeddings of a real-world static CIFAR dataset arranged into a stream to simulate concept drifts and the emergence of unknowns, as well as real-world datasets where novelty is observed. The methodology was compared with state-of-the-art approaches with three-criterion recognition quality assessment and resource consumption evaluation. The results indicate that the proposed method is competitive with reference approaches in stationary environments and performs best when concept drifts occur.
Figures & tables
Fig. 1: An example of a non-stationary data stream, where, over 100 chunks, four novel classes appear (beige markers). The known class distribution (blue and red markers) shifts due to concept drifts. The bottom plot shows the proportions of classes at a given timestamp.
Fig. 2: The normalized distributions of reconstruction errors of known classes (black bars) and novel-class samples (blue bars). The central column shows the responses of KDE (red color indicates high density score) and the right column the label known/unknown produced by orcca . Scatterplots use two principal components.
Fig. 3: The error distributions of two known classes when samples are reconstructed by both autoencoders. The black bars indicate errors of a given class, and blue bars – errors of the other class. The scatterplots show the distribution of errors with thresholds (dashed lines) and the principal components with labels on the last plot. The three samples that fell outside of both thresholds were marked as unknowns (black points).
comparison
resource use
hyperparameters
tabular
cifar-10
tabular
source distribution
Madelon
ResNet-18
Madelon
features (informative)
100 (50)
512
100 (50)
samples
5,000
5,000
100,000
known + unknown classes
2+8, 4+6, 6+2
2+8, 4+6, 6+2
3+0, 3+3, 3+5
concept drifts
0, 1, 3, 5, 7
0, 1, 3, 5, 7
0, 4
TABLE I: The configuration of synthetic / semi-synthetic data stream generation used in the experiments.
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.974
0.997
0.961
0.927
0.184
0.815
—
0.964
K2 N8 D1
0.975
0.979
0.908
0.927
0.185
0.807
—
0.966
K2 N8 D3
0.986
0.679
0.639
0.936
0.206
0.817
—
0.973
K2 N8 D5
0.991
0.470
0.446
0.95
0.25
0.75
—
0.990
K2 N8 D7
0.991
0.359
0.309
0.931
0.28
0.755
—
0.980
K4 N6 D0
0.966
0.992
0.97
0.72
0.182
0.806
—
0.908
TABLE II: Results of known class balanced accuracy (criterion 1) metric for tabular data streams (CD=2.33).
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.933
0.831
0.706
0.505
0.498
0.769
0.705
0.509
K2 N8 D1
0.947
0.833
0.680
0.501
0.504
0.767
0.674
0.511
K2 N8 D3
0.982
0.689
0.544
0.496
0.499
0.764
0.579
0.511
K2 N8 D5
0.989
0.590
0.448
0.490
0.499
0.783
0.536
0.510
K2 N8 D7
0.991
0.536
0.378
0.480
0.506
0.798
0.511
0.505
K4 N6 D0
0.893
0.794
0.581
0.501
0.500
0.536
0.641
0.508
TABLE III: Results of known vs unknown class balanced accuracy (criterion 2) metric for tabular data streams (CD=2.71).
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.443
0.276
0.051
0.033
0.026
0.611
—
—
K2 N8 D1
0.458
0.295
0.049
0.039
0.026
0.617
—
—
K2 N8 D3
0.489
0.304
0.047
0.038
0.028
0.526
—
—
K2 N8 D5
0.483
0.313
0.039
0.033
0.028
0.479
—
—
K2 N8 D7
0.495
0.345
0.041
-0.002
0.028
0.442
—
—
K4 N6 D0
0.378
0.317
0.070
0.19
0.024
0.579
—
—
TABLE IV: Results of unknown class ARI (criterion 3) metric for tabular streams (CD=1.95).
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.732
0.981
0.962
0.952
0.424
0.434
—
0.799
K2 N8 D1
0.720
0.980
0.932
0.955
0.421
0.436
—
0.805
K2 N8 D3
0.819
0.893
0.735
0.947
0.432
0.466
—
0.864
K2 N8 D5
0.924
0.732
0.553
0.937
0.429
0.559
—
0.889
K2 N8 D7
0.964
0.569
0.410
0.926
0.408
0.695
—
0.922
K4 N6 D0
0.548
0.881
0.796
0.630
0.258
0.408
—
0.387
TABLE V: Results of known class balanced accuracy (criterion 1) metric for CIFAR data streams (CD=2.33).
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.575
0.551
0.579
0.503
0.549
0.630
0.653
0.488
K2 N8 D1
0.575
0.550
0.564
0.499
0.563
0.631
0.620
0.489
K2 N8 D3
0.737
0.511
0.462
0.492
0.556
0.607
0.521
0.492
K2 N8 D5
0.921
0.429
0.367
0.484
0.555
0.563
0.479
0.493
K2 N8 D7
0.971
0.345
0.295
0.478
0.593
0.566
0.452
0.500
K4 N6 D0
0.517
0.518
0.529
0.501
0.484
0.532
0.597
0.472
TABLE VI: Results of known vs unknown class balanced accuracy (criterion 2) for CIFAR data streams (CD=2.71).
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
K2 N8 D0
0.167
0.153
0.152
0.142
0.059
0.306
—
—
K2 N8 D1
0.159
0.153
0.152
0.145
0.059
0.305
—
—
K2 N8 D3
0.084
0.156
0.149
0.132
0.056
0.316
—
—
K2 N8 D5
0.075
0.156
0.150
0.069
0.058
0.315
—
—
K2 N8 D7
0.101
0.155
0.152
0.058
0.068
0.319
—
—
K4 N6 D0
0.249
0.255
0.238
0.215
0.067
0.368
—
—
TABLE VII: Results of unknown class ARI (criterion 3) metric for CIFAR streams (CD=1.95).
Fig. 4: Averaged results from comparison experiment on synthetic and semi-synthetic data streams (tabular and based on CIFAR) in all three criteria, grouped in pairs.
Fig. 5: The presentation of ranks of all evaluated methods and the critical differences (dashed lines) in all three criteria, grouped in pairs. The best methods will appear in top-right corner of plots.
stream
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
known class balanced accuracy (criterion 1)
covertype
0.202
0.141
0.394
0.325
0.207
0.341
—
0.330
kdd
—
—
—
—
—
—
—
—
pamap2
—
—
—
—
—
—
—
—
mean rank
6
7
1
4
5
2
—
3
known vs unknown class balanced accuracy (criterion 2)
TABLE VIII: Results of quality metrics for real-world data streams in all three criteria. For KDD and PAMAP2 only one class was visible in the offline phase, making it impossible to measure performance in criterion 1.
blocks
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
stream: 0 novelties, 0 drifts
start
435.2
7.7
43.3
1537.0
3931.1
102.2
42.4
498.6
end
434.4
7.8
43.1
1558.9
3882.2
102.0
42.1
506.4
% growth
-0.2
0.4
-0.5
1.4
-1.2
-0.1
-0.5
1.6
stream: 0 novelties, 4 drifts
start
437.0
8.6
66.7
1431.4
3797.6
102.1
42.1
531.9
TABLE IX: Processing time per sample (in μs ) by the proposed and the reference methods.
blocks
ORCCA
MINAS
ECSMin
ECHO
CluStr
DenStr
OCND
CND
stream: 0 novelties, 0 drifts
start
663.8
19.1
30.3
43.1
996.3
69.6
159.3
329.3
end
690.1
19.0
30.1
43.2
999.6
69.6
159.3
329.3
% growth
4.0
-0.6
-0.6
0.4
0.3
0.0
0.0
0.0
stream: 0 novelties, 4 drifts
start
657.7
21.6
33.5
43.9
1005.4
69.6
159.3
329.3
TABLE X: Results of memory usage (in kB ) by the proposed and the reference methods.
Fig. 6: The visual presentation of time and memory resources growth. The points indicate the increase between initial processing steps and at the end of a stream. Red markers indicate results for ORCCA. Its processing time growth is minimal, and its memory growth is lower than that of most reference methods.
stream
ORCCA
no adapt
no weight
no split
cascade
kde
known class balanced accuracy (criterion 1)
K2 N8 D0
0.974
-0.004
+0.001
0.000
-0.024
-0.097
K2 N8 D5
0.991
-0.218
0.000
0.000
-0.010
-0.042
K4 N6 D0
0.966
-0.006
+0.002
0.000
-0.067
-0.121
K4 N6 D5
0.986
-0.274
+0.001
0.000
-0.025
-0.152
Covertype
0.202
+0.025
+0.014
0.000
-0.010
-0.025
TABLE XI: Ablation study results. Positive values indicate a benefit of a given mechanism, the negative values indicate a reduction of recognition quality in relation to the ORCCA baseline.
Data stream processing has become a landmark in modern machine learning applications, with concept drifts and novel class appearances posing the primary challenges faced by sophisticated recognition methods. This work proposes an unsupervised concept drift detection method that identifies shifts in known class distributions based on the reconstruction errors of an autoencoder, while also enabling the recognition of novel class samples through density estimation of a proxy representation of samples. Using mirrored autoencoders allows for independent incremental adaptation to changing problem distributions for the two considered tasks, resulting in continuous adjustment to evolving concepts and reliable recognition of unknown samples. Conducted experiments used a diverse set of synthetic tabular data streams, where both concept drifts and the emergence of novelties were observed. The results show that the proposed approach is competitive with current state-of-the-art unsupervised drift detectors and novelty classifiers.
Joanna Komorniczak
Department of Systems and Computer Networks, Wrocław University of Science and Technology
Traditional machine learning assumes a stationary data distribution, yet many real-world applications operate on nonstationary streams in which the underlying concept evolves over time. This problem can also be viewed as task-free continual learning under concept drift, where a model must adapt sequentially without explicit task identities or task boundaries. In such settings, effective learning requires both rapid adaptation to new data and forgetting of outdated information. A common solution is based on a sliding window, but this approach is often computationally demanding because the model must be repeatedly retrained from scratch on the most recent data. We propose a different perspective based on machine unlearning. Instead of rebuilding the model each time the active window changes, we remove the influence of outdated samples using unlearning and then update the model with newly observed data. This enables efficient, targeted forgetting while preserving adaptation to evolving distributions. To the best of our knowledge, this is the first work to connect machine unlearning with concept drift mitigation for task-free continual learning. Empirical results on image stream classification across multiple drift scenarios demonstrate that the proposed approach offers a competitive and computationally efficient alternative to standard sliding-window retraining. Our implementation can be found at \hrehttps://anonymous.4open.science/r/MUNDataStream-60F3}{https://anonymous.4open.science/r/MUNDataStream-60F3}.
Michal Wozniak, Marek Klonowski, Maciej Maczynski +1
Wroclaw University of Science and Technology Wrocław, Poland · Rochester Institute of Technology NY, USA
Machine learning algorithms deployed for evolving streaming environments must handle the non-stationary data distributions, commonly referred to as concept drift. The presence of concept drift poses a major challenge for many real-world applications because it can severely degrade their predictive performance, hindering their ability to support robust decision-making. Consequently, the timely and efficient detection of drift events is critical for sustaining high accuracy over time. This study examines theoretically the concept drift characteristics and numerous drift detection algorithms across several categories. Furthermore, we evaluate their performance on both synthetic and real-world datasets exhibiting diverse streaming scenarios and drift characteristics, such as abrupt and gradual changes. This study aims to enhance understanding of the complex notion of concept drift characteristics and behavior of drift detectors, along with their applicability to diverse contexts.
Md Moman Ul Haque Khan, Samira Sadaoui
Department of Computer Science, University of Regina, Canada