Internet of Things (IoT) Analytics often involves applying machine learning (ML) models on data streams. In such scenarios, traditional ML paradigms face obstacles related to continuous learning while dealing with concept drifts, temporal dependence, and avoiding forgetting. Moreover, in IoT, different edge devices build up a network. When learning models on those devices, connecting them could be useful in improving performance and reusing others' knowledge. This work proposes Mutual Assisted Learning, a learning paradigm grounded on Vygotsky's popular Sociocultural Theory of Cognitive Development. Each device is autonomous and does not need a central orchestrator. Whenever it degrades its performance due to a concept drift, it asks for assistance from others and decides whether their knowledge is useful for solving the new problem. This way, the number of connections is drastically reduced compared to the classical Federated Learning approaches, where the devices communicate at each training round. Every device is equipped with a Continuous Progressive Neural Network (cPNN) to handle the dynamic nature of data streams. We call this implementation Mutual Assisted cPNN (MAcPNN). To implement it, we allow cPNNs for single data point predictions and apply quantization to reduce the memory footprint. Experimental results prove the effectiveness of MAcPNN in boosting performance on synthetic and real data streams.
Figures & tables
Figure 1: Size of cPNN and QcPNN ( 1 ) and QcPNN Compression Ratio ( 1 ) considering data points with 10 features. Models are implemented using PyTorch, an LSTM hidden size of 50, and a window size of 10. INT8 quantization is applied.
Figure 2: MAL paradigm. When U1 detects a drift on its data stream DS1 , it asks for assistance from the other devices. U2 and U3 send to U1 copies of their local models. U1 builds an ensemble containing its local models and copies of the others. U1 adds a column to all the models. Grey represents the frozen columns, while red represents the trainable columns.
Figure 3: Concept organization within the data streams of three users’ devices. We combine five classification functions with abrupt drifts: some concepts are seen by others, and some are new. We boldly highlight concepts seen by other devices. Drifts are unsynchronized across streams.
Figure 4: An example of Cohen’s Kappa evolution over time of Device 2 on a Weather configuration to highlight the points in which we observe start and end metrics. Scores are reset after each drift. The score at data point dt considers the predictions from the first data point following the last drift to dt .
start 1
start 2
start 3
start 4
start avg
end 1
end 2
end 3
end 4
end avg
S R W
ARF T
0.74±0.00
0.74±0.00
0.73±0.00
0.73±0.00
0.73±0.00
0.73±0.00
0.73±0.00
0.73±0.00
0.73±0.00
0.73±0.00
cLSTM
0.69±0.05
0.69±0.05
0.65±0.06
0.73±0.05
0.69±0.03
0.80±0.04
0.81±0.04
0.78±0.06
0.84±0.03
0.81±0.02
cPNN
0.67±0.03
0.72±0.06
0.75±0.07
0.71±0.02
0.71±0.02
0.80±0.02
0.81±0.04
0.81±0.06
0.78±0.02
0.80±0.02
MAcPNN
0.74±0.06
0.84±0.06
0.80±0.06
0.85±0.04
0.81±0.03
0.83±0.03
0.88±0.04
0.85±0.06
0.88±0.03
0.86±0.02
W
ARF T
0.35±0.12
0.42±0.15
0.43±0.21
0.43±0.08
0.41±0.04
0.40±0.12
0.45±0.15
0.44±0.21
0.46±0.07
0.44±0.04
cLSTM
0.47±0.06
0.55±0.09
0.48±0.13
0.62±0.06
0.53±0.03
0.62±0.05
0.67±0.07
0.62±0.11
0.70±0.04
0.65±0.02
Table I : Cohen’s Kappa average scores and the corresponding standard deviations for 10 SRW configurations and 50 Weather (W) and AirQuality (AQ) configurations. Given a configuration, for each concept drift j , we compute the mean values of start j and end j over the three devices of the network. start avg and end avg are computed by first averaging the different start j and end j values for each device separately and then averaging these three values. We then compute the mean and the standard deviation over all the configurations of each type of data stream (SRW, W, and AQ). We highlight the statistically best-performing model in bold. MAcPNN is always the best model after the beginning of the concepts. After the concepts end, cLSTM can approach MAcPNN only on Weather.
start 1
start 2
start 3
start 4
start avg
end 1
end 2
end 3
end 4
end avg
S R W
ARF T
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
0.87±0.00
cLSTM
0.84±0.02
0.84±0.02
0.82±0.03
0.87±0.02
0.84±0.02
0.90±0.02
0.90±0.02
0.89±0.03
0.92±0.01
0.90±0.01
cPNN
0.83±0.02
0.86±0.03
0.87±0.04
0.85±0.01
0.86±0.01
0.90±0.01
0.91±0.02
0.91±0.03
0.89±0.01
0.90±0.01
MAcPNN
0.87±0.03
0.92±0.03
0.90±0.03
0.93±0.02
0.90±0.01
0.92±0.02
0.94±0.02
0.92±0.03
0.94±0.02
0.93±0.01
W
ARF T
0.68±0.06
0.71±0.08
0.71±0.11
0.72±0.04
0.70±0.02
0.70±0.06
0.72±0.07
0.72±0.11
0.73±0.04
0.72±0.02
cLSTM
0.73±0.03
0.77±0.05
0.74±0.07
0.81±0.03
0.76±0.01
0.81±0.03
0.83±0.04
0.81±0.05
0.85±0.02
0.83±0.01
Table II : Balanced Accuracy average scores and the corresponding standard deviations for 10 SRW configurations and 50 Weather (W) and AirQuality (AQ) configurations. MAcPNN is always the best model after the beginning of the concepts. After the concepts end, cLSTM can approach MAcPNN only on Weather.
Figure 5: Critical distance diagram of the Nemenyi test ( α=0.05 ) on 100 real data streams (50 of Weather and 50 of AirQuality) on the different start j and start avg considering the Cohen’s Kappa score. A is ARF T , cL is cLSTM, M is MAcPNN, and cP is cPNN. MAcPNN is statistically the best model in all the cases. Applying MAcPNN boosts the adaptation to new concepts.
Figure 6: Critical distance diagram of the Nemenyi test ( α=0.05 ) on 100 real data streams (50 of Weather and 50 of AirQuality) on the different end j and end avg considering the Cohen’s Kappa score. A is ARF T , cL is cLSTM, M is MAcPNN, and cP is cPNN. MAcPNN is statistically the best model in all the cases (except for end 4 where it still has the best rank). Applying MAcPNN improves the performance of the whole concept.
Figure 7: Average Cohen’s Kappa over time evolution on the network on 50 configurations of Weather ( 7 ) and AirQuality ( 7 ). For each configuration, we first average the scores of the different concepts (from the second onwards) on each device separately. Then, we average the scores across the three devices, and finally, we average these scores over all configurations. MAcPNN adapts better to concept drifts. On Weather, cLSTM can approach MAcPNN performance at the end of the concept. On AirQuality, MAcPNN keeps the best models over the entire concept.
Concept drift, temporal dependence, and catastrophic forgetting represent major challenges when learning from data streams. While Streaming Machine Learning and Continual Learning (CL) address these issues separately, recent efforts in Streaming Continual Learning (SCL) aim to unify them. In this work, we introduce MAGIC Net, a novel SCL approach that integrates CL-inspired architectural strategies with recurrent neural networks to tame temporal dependence. MAGIC Net continuously learns, looks back at past knowledge by applying learnable masks over frozen weights, and expands its architecture when necessary. It performs all operations online, ensuring inference availability at all times. Experiments on synthetic and real-world streams show that it improves adaptation to new concepts, limits memory usage, and mitigates forgetting.
Federico Giannini, Sandro D'Andrea, Emanuele Della Valle
Dealing with an unbounded data stream involves overcoming the assumption that data is identically distributed and independent. A data stream can, in fact, exhibit temporal dependencies (i.e., be a time series), and data can change distribution over time (concept drift). The two problems are deeply discussed, and existing solutions address them separately: a joint solution is absent. In addition, learning multiple concepts implies remembering the past (a.k.a. avoiding catastrophic forgetting in Neural Networks' terminology). This work proposes Continuous Progressive Neural Networks (cPNN), a solution that tames concept drifts, handles temporal dependencies, and bypasses catastrophic forgetting. cPNN is a continuous version of Progressive Neural Networks, a methodology for remembering old concepts and transferring past knowledge to fit the new concepts quickly. We base our method on Recurrent Neural Networks and exploit the Stochastic Gradient Descent applied to data streams with temporal dependencies. Results of an ablation study show a quick adaptation of cPNN to new concepts and robustness to drifts.
Federico Giannini, Giacomo Ziffer, Emanuele Della Valle
In this work, we present OCLADS, a novel communication framework with continual learning (CL) for Internet of Things (IoT) anomaly detection (AD) when operating in non-stationary environments. As the statistical properties of the observed data change with time, the on-device inference model becomes obsolete, which necessitates strategic model updating. OCLADS keeps track of data distribution shifts to timely update the on-device IoT AD model. To do so, OCLADS introduces two mechanisms during the interaction between the resource-constrained IoT device and an edge server (ES): i) an intelligent sample selection mechanism at the device for data transmission, and ii) a distribution-shift detection mechanism at the ES for model updating. Experimental results with TinyML demonstrate that our proposed framework achieves high inference accuracy while realizing a significantly smaller number of model updates compared to the baseline schemes.
Faculty of Electrical Engineering and Information Technologies, Ss. Cyril and Methodius University in Skopje, 1000 Skopje, North Macedonia · Dept. of Electronic Systems, Aalborg University, Aalborg, Denmark