Performance Drift Detection in Machine Learning as a Service (MLaaS) for IoT Environments
Authors: Deepak Kanneganti, Sajib Mistry, Sheik Mohammad Mostakim Fattah, Erik Elmroth, Aneesh Krishna, Monowar Bhuyan
Organizations: School of Electrical Engineering, Computing and Mathematical Sciences (EECMS), Curtin University, Perth, WA, Australia · Department of Computing Science, Umeå University, Umeå, Sweden
Machine Learning as a Service (MLaaS) is a powerful cloud paradigm enabling data-driven intelligent applications in Internet of Things (IoT) environments, widely adopted across healthcare, smart homes, and industry due to its cost-effectiveness. However, the dynamic nature of IoT frequently alters data distributions, affecting MLaaS stability, while periodic MLaaS updates further introduce performance drift. Unlike traditional ML systems, MLaaS clients operate as black-box users without access to internal data or parameters, making drift detection particularly challenging. To address this, we propose a novel MLaaS Performance Drift Detection framework for IoT environments. The framework first employs an MLaaS extraction model that learns service behavior from input-output pairs and identifies prediction-influenced features. Building on this, the proposed MLaaS Performance Drift Detection (MPDD) model jointly captures variations in input data and MLaaS behavior. We further design an Adaptive-Temporal Performance Drift Detection Mechanism (APDDM) that dynamically adjusts monitoring frequency based on behavioral and data variations, enabling timely drift detection for effective service management. Extensive experiments on real-world datasets demonstrate that MPDD achieves up to 22-25% accuracy improvement over baseline drift detection methods. APDDM provides an average accuracy gain of approximately 4% and reduces the miss detection rate by around 9% compared to fixed-interval monitoring.
Figures & tables
Fig. 1: Illustration of the Motivation Scenario
Notation
Description
M
Black-box ML service (MLaaS model).
F
Functional specification of the ML service.
QoS
Non-functional (quality) attributes of the service.
Ds
Data specification of the service.
X
Input feature vector.
d
Number of input features.
TABLE I: Notation Summary
Fig. 2: Machine Learning as a Service (MLaaS) Performance Drift Detection Framework for IoT environments
Parameter
Configuration
Parameter
Configuration
Model
CART decision tree
Criterion
Gini impurity
Split strategy
Best split
Maximum depth
4
Min samples split
2
Min samples leaf
1
Maximum features
All variables
Training target
MLaaS labels
Train/Test split
50% / 50%
Training data
60% clean + 40% drift
TABLE II: Configuration of the MLaaS extraction model M′ .
Dataset
Query Pool
Clean
Drift
Train
Test
HAR
100,000
60,000
40,000
50,000
50,000
Electricity
75,000
45,000
30,000
37,500
37,500
Weather
160,000
96,000
64,000
80,000
80,000
Poker
160,000
96,000
64,000
80,000
80,000
Airline
16,649
9,989
6,660
8,324
8,325
TABLE III: Training data configuration of the MLaaS extraction model
Method
Drift
HAR [ 34 ]
Electricity [ 35 ]
Weather [ 36 ]
Poker [ 38 ]
Airline [ 37 ]
Acc
Prec
Rec
Acc
Prec
Rec
Acc
Prec
Rec
Acc
Prec
Rec
Acc
Prec
Rec
MPDD
Pseudo
0.91
0.88
0.98
0.90
0.87
0.98
0.89
0.85
0.98
0.88
0.89
0.94
0.90
0.87
0.98
Real
0.98
0.71
0.99
0.72
0.98
0.71
0.85
0.75
0.98
0.69
ADWIN [ 18 ]
Pseudo
0.62
0.66
0.74
0.77
0.77
0.83
0.50
0.46
0.65
0.54
0.61
0.64
0.65
0.71
0.71
Real
0.53
0.44
0.77
0.69
0.57
0.38
0.42
0.39
0.56
0.56
MMD [ 26 ]
Pseudo
0.52
0.59
0.64
0.69
0.69
0.82
0.54
0.49
0.70
0.54
0.61
0.65
0.65
0.71
0.71
TABLE IV: Comparison of MLaaS Performance Drift Detection (MPDD) under pseudo drift and real drift across datasets.
Fig. 3: Comparative accuracy analysis of drift detection techniques across datasets: (a) HAR, (b) Electricity, (c) Weather, (d) Poker, (e) Airline, and (f) threshold sensitivity analysis, evaluated using the proposed MPDD model.
Fig. 4: Timeliness evaluation of APDDM across datasets: (a) HAR, (b) Electricity, (c) Weather, (d) Airline, and (e) Poker, showing accuracy comparisons for each scenario.
Fig. 5: False negative count comparison across the HAR, Airline, Electricity, Weather, and Poker datasets.
Data
Method
Acc. ↑
Acc. Gain ↑
MDR ↓
FP ↓
FN ↓
HAR
ADWIN
0.62
30.42
0.56
190
270
MMD
0.52
40.00
0.66
260
315
D3
0.30
62.50
0.73
495
350
STUDD
0.69
23.23
0.96
30
1465
SCSD
0.65
26.98
0.02
1650
25
MPDD
0.91
1.25
0.29
0
440
TABLE V: Detection-quality comparison across drift detection techniques.
Fig. 6: Fidelity and predictive behavior analysis of the MLaaS extraction model across datasets: (a) predictive accuracy comparison, and (b) predictive behavior analysis
Fig. 7: Comparison of MPDD and APDDM accuracy when using the MLaaS extracted model and the original MLaaS
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
No Drift
Sudden
Incremental
Gradual
Recurrent
Total Drift
HAR
60,000
24,000
14,400
14,000
24,000
76,400
Airline
9,989
3,996
2,398
2,332
3,998
12,724
Electricity
45,312
20,391
12,235
11,329
18,126
62,081
Weather
96,453
53,050
31,830
24,109
38,583
147,572
Poker
100,000
40,000
24,000
23,336
40,000
127,336
Appendix
TABLE 6: Drift generation statistics across datasets.
Fig. 8: Drift generation across datasets: (A) HAR, (B) Airline, (C) Weather, (D) Electricity, and (E) Poker.
School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia · KDDI Research, Japan · School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia