Dealing with an unbounded data stream involves overcoming the assumption that data is identically distributed and independent. A data stream can, in fact, exhibit temporal dependencies (i.e., be a time series), and data can change distribution over time (concept drift). The two problems are deeply discussed, and existing solutions address them separately: a joint solution is absent. In addition, learning multiple concepts implies remembering the past (a.k.a. avoiding catastrophic forgetting in Neural Networks' terminology). This work proposes Continuous Progressive Neural Networks (cPNN), a solution that tames concept drifts, handles temporal dependencies, and bypasses catastrophic forgetting. cPNN is a continuous version of Progressive Neural Networks, a methodology for remembering old concepts and transferring past knowledge to fit the new concepts quickly. We base our method on Recurrent Neural Networks and exploit the Stochastic Gradient Descent applied to data streams with temporal dependencies. Results of an ablation study show a quick adaptation of cPNN to new concepts and robustness to drifts.
Figures & tables
Figure 1: Comparison of CL and SML scenarios.
Figure 2: Loss functions’ minimum and accuracy trend of a single linear neuron associated with the following classification functions: (a) −x1+x2−0.8≥0 (b) −x1+x2−0.7≥0 (c) −x1+x2−0.7<0 .
Figure 3: Data processing in cases of Traditional Machine Learning and SML.
Figure 4: The cPNN architecture during the second concept training.
concept
case
cPNN
cLSTM
mcLSTM
cPNN
cLSTM
mcLSTM
[1,50]
.96, .004
.872, .026
.903, .008
.864, .017
.742, .024
.75, .006
[1,100]
.972, .002
.921, .014
.933, .004
.888, .02
.79, .024
.774, .006
(100,)
.989, .001
.98, .002
.975, .001
.927, .018
.883, .029
.834, .009
2nd
[1,)
.984, .001
.965, .004
.964, .001
.917, .018
.859, .026
.818, .007
Table 1: Accuracies on classification inversion drift. cPNN outperforms the ablated versions in all cases.
concept
case
cPNN
cLSTM
mcLSTM
cPNN
cLSTM
mcLSTM
[1,50]
.759, .012
.746, .011
.738, .006
.781, .006
.743, .015
.753, .006
[1,100]
.803, .007
.791, .012
.762, .005
.808, .004
.772, .008
.775, .004
(100,)
.877, .006
.897, .013
.829, .011
.876, .005
.851, .011
.823, .009
2nd
[1,)
.858, .005
.87, .012
.812, .009
.859, .003
.831, .009
.811, .007
[1,50]
.951, .004
.893, .022
.905, .008
.947, .004
.94, .009
.908, .004
Table 2: Accuracies on data streams , , , and , , , . cPNN always recovers faster from concept drifts than the ablated versions. In some cases, a single cLSTM performs better in the long run, but in the end, it only remembers that last concept since it does not manage CF. mcLSTM that does not use transfer learning and resets the parameter configuration performs worse in almost all situations.
concept
case
cPNN
cLSTM
mcLSTM
cPNN
cLSTM
mcLSTM
[1,50]
. 931, .014
.933, .012
.915, .004
.921, .008
.892, .022
.903, .003
[1,100]
. 943, .01
.951, .008
.941, .003
. 939, .009
.934, .012
.934, .003
(100,)
.974, .003
.983, .001
.976, .001
.971, .004
.983, .001
.975, .001
2nd
[1,)
.966, .005
.974, .003
.967, .001
. 963, .005
.97, .003
.964, .001
[1,50]
.824, .013
.768, .023
.754, .005
.835, .013
.779, .016
.755, .007
Table 3: Accuracies on data streams , , , and , , , .
Concept drift, temporal dependence, and catastrophic forgetting represent major challenges when learning from data streams. While Streaming Machine Learning and Continual Learning (CL) address these issues separately, recent efforts in Streaming Continual Learning (SCL) aim to unify them. In this work, we introduce MAGIC Net, a novel SCL approach that integrates CL-inspired architectural strategies with recurrent neural networks to tame temporal dependence. MAGIC Net continuously learns, looks back at past knowledge by applying learnable masks over frozen weights, and expands its architecture when necessary. It performs all operations online, ensuring inference availability at all times. Experiments on synthetic and real-world streams show that it improves adaptation to new concepts, limits memory usage, and mitigates forgetting.
Federico Giannini, Sandro D'Andrea, Emanuele Della Valle
Most contemporary neural learning systems rely on epoch-based optimization and repeated access to historical data, implicitly assuming reversible computation. In contrast, real-world environments often present information as irreversible streams, where inputs cannot be replayed or revisited. Under such conditions, conventional architectures degrade into reactive filters lacking long-horizon coherence. This paper introduces Stream Neural Networks (StNN), an execution paradigm designed for irreversible input streams. StNN operates through a stream-native execution algorithm, the Stream Network Algorithm (SNA), whose fundamental unit is the stream neuron. Each stream neuron maintains a persistent temporal state that evolves continuously across inputs. We formally establish three structural guarantees: (1) stateless mappings collapse under irreversibility and cannot encode temporal dependencies; (2) persistent state dynamics remain bounded under mild activation constraints; and (3) the state transition operator is contractive for λ < 1, ensuring stable long-horizon execution. Empirical phase-space analysis and continuous tracking experiments validate these theoretical results. The execution principles introduced in this work define a minimal substrate for neural computation under irreversible streaming constraints.
Internet of Things (IoT) Analytics often involves applying machine learning (ML) models on data streams. In such scenarios, traditional ML paradigms face obstacles related to continuous learning while dealing with concept drifts, temporal dependence, and avoiding forgetting. Moreover, in IoT, different edge devices build up a network. When learning models on those devices, connecting them could be useful in improving performance and reusing others' knowledge. This work proposes Mutual Assisted Learning, a learning paradigm grounded on Vygotsky's popular Sociocultural Theory of Cognitive Development. Each device is autonomous and does not need a central orchestrator. Whenever it degrades its performance due to a concept drift, it asks for assistance from others and decides whether their knowledge is useful for solving the new problem. This way, the number of connections is drastically reduced compared to the classical Federated Learning approaches, where the devices communicate at each training round. Every device is equipped with a Continuous Progressive Neural Network (cPNN) to handle the dynamic nature of data streams. We call this implementation Mutual Assisted cPNN (MAcPNN). To implement it, we allow cPNNs for single data point predictions and apply quantization to reduce the memory footprint. Experimental results prove the effectiveness of MAcPNN in boosting performance on synthetic and real data streams.