Organizations: School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia · KDDI Research, Japan · School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia
Identification of IoT device types from passive traffic is increasingly used for security management in enterprise and ISP networks. However, the performance of machine learning-based classifiers gradually degrades under concept drift as device behavior evolves. Therefore, maintaining classification performance requires periodic retraining with newly labeled deployment traffic. The operational challenge is determining how much and which deployment traffic instances to label for maintaining classification performance. We show that these two decisions should be treated separately. While retraining solely on instances selected by a drift detector is prone to systematically overlooking parts of the emerging behavioral space, uniformly sampled deployment traffic captures more representative behavioral changes. Instead, drift detection is more effective at determining the amount of deployment traffic that should be labeled. We make three contributions. (1) We conduct a two-year longitudinal study of IoT traffic and characterize how behavioral evolution manifests across device classes and how retraining with newly labeled traffic restores classification performance. (2) We develop a conformity-based drift detector that captures class-conditional behavioral models directly from raw traffic features and provides feature-level explanations of behavioral evolution. (3) We demonstrate that adjusting the traffic labeling rate according to the observed behavioral evolution, combined with uniform traffic sampling, maintains classifier performance more effectively than detector-guided sample selection and is beneficial to managing the traffic labeling effort. We further show that this strategy performs comparably to confidence-guided adaptation while providing feature-level explanations. Our evaluation uses 3.8 million IPFIX flow records collected from 21 IoT types over more than 2 years.
Figures & tables
Device Make-and-Model
Category
Amazon Echo Show 5
Voice assistant
Amazon Echo Gen 2
Voice assistant
Google Nest Mini
Voice assistant
Amazon Echo Dot with Clock
Voice assistant
Google Home
Voice assistant
Sony Smart Speaker
Voice assistant
TABLE I : The 21 IoT devices studied.
Feature Name
Description
Abbr. Name (reverse)
packetTotalCount
# of packets.
pktTotCnt (rPktTotCnt)
octetTotalCount
# of bytes (header and payload).
octTotCnt (rOctTotCnt)
smallPacketCount
# of packets with less than 60 Bytes
of payload.
sPktCnt (rSPktCnt)
largePacketCount
# of packets with at least 220 Bytes
of payload.
lPktCnt (rLPktCnt)
TABLE II : Traffic features in IPFIX flow records.
Fig. 1 : Behavioral evolution during long-term deployment progressively degrades classifier performance. The partial restoration observed in late 2023 suggests that device behaviors may evolve non-monotonically away from the training distribution.
Fig. 2 : Behavioral evolution manifests differently across IoT device classes. Representative examples show persistent behavior (a), behavioral decay (b), and behavioral restoration (c), while the corresponding t-SNE projection (d) illustrates how changes in traffic distribution explain these different evolution patterns. The black curves show the upper-bound maintenance performance obtained by retraining with all newly labeled deployment traffic.
Fig. 3 : Upper-bound maintenance performance established by retraining the classifier using exhaustively labeled deployment traffic. This experiment demonstrates that long-term classifier maintenance is feasible, but highlights the prohibitive labeling effort required in practice.
Fig. 4 : Feature-level view of a correctly classified deployment window for Philips Hue Bridge. Although the observed feature values (red triangles) occupy only a subset of the training observations (gray markers), they remain within previously learned behavioral patterns, illustrating behavioral conformity despite distributional differences.
Fig. 5 : Drifted instances reveal feature-level non-conformity with respect to the predicted class. Even when both flows are misclassified as Apple HomePod, different features expose their deviation from the training distribution.
Fig. 6 : Modeling the behavioral range of non-negative traffic features. (a) GKDE assigns non-zero probability to infeasible feature values ( x<0 ), which is invalid for non-negative features. (b) Log-normal KDE (LNKDE) preserves non-negativity while accurately modeling the observed behavioral range of the feature.
Fig. 7 : Construction of feature-level conformity regions. The conformity threshold θ is chosen so that the probability mass of the non-conforming region (shaded) does not exceed the user-defined sensitivity parameter ϵ .
Fig. 8 : Relationship between behavioral non-conformity and classifier misclassification. Mean TPR (a) and specificity (b) are reported for each device class over the deployment phase. Error bars indicate temporal standard deviation across evaluation windows.
Fig. 9 : Temporal evolution of detector performance averaged across device classes. TPR remains relatively stable throughout deployment, whereas specificity gradually decreases as behavioral evolution introduces an increasing number of correctly classified instances exhibiting previously unseen patterns.
ϵ
TPR
Specificity
Balanced accuracy
0.001
0.65±0.27
0.58±0.31
0.61±0.17
0.010
0.77±0.22
0.49±0.27
0.63±0.15
0.100
0.95±0.08
0.26±0.19
0.60±0.10
TABLE III : Detection results for different ϵ values.
Fig. 10 : Feature-level explanations of behavioral evolution for the Wansview Q5 IP camera. Misclassifications across two deployment periods are explained by distinct subsets of non-conforming traffic features.
Fig. 11 : (a) Classification performance under different adaptation strategies. Detector-guided adaptation (solid green) substantially improves performance over no adaptation (dotted blue), but remains consistently below the upper-bound maintenance strategy (dashed gray). Replacing detector-selected retraining instances with uniformly sampled deployment traffic at the same detector-guided labeling rate (dash-dotted green) yields performance close to the upper bound. (b) Fraction of deployment traffic labeled in each evaluation window. Detector-guided adaptation labels progressively fewer instances over time as fewer deployment flows are identified as behaviorally non-conforming, whereas uniform sampling preserves the detector-guided sampling rate while providing more representative coverage of the evolving behavioral patterns. For the strategy involving random sampling, error bars indicate the standard deviation across 10 random seeds.
Fig. 12 : Detector-selected instances (red crosses) concentrate around only a subset of the emerging behavioral patterns (orange circles). Uniformly sampled instances (black stars) provide substantially broader coverage of the evolving traffic behavior, explaining their superior adaptation performance.
Fig. 13 : Classifier maintenance under a fixed operational labeling budget. Sampling uniformly at the full budget (dash-dotted pink) provides a reference operating point. Reducing the sampling rate uniformly across deployment (dotted orange) noticeably degrades adaptation performance. In contrast, adjusting the sampling rate according to the observed behavioral evolution (solid maroon) achieves comparable performance while labeling fewer deployment instances. Error bars are omitted for visual clarity.
Metric
Cap. rate
Uniform below cap.
Behavior-aware rate
Low-confidence
Low-confidence stratified
Confidence-guided rate
Avg. macro F1-score
0.8970±0.0042
0.8797±0.0067
0.8934±0.0049
0.8845
0.8905
0.8942±0.0039
Labeling cost
0.1%(Cap)
0.0501±0.0022%
0.0826±0.0006%
0.1%(Cap)
0.123%
0.0867±0.0009%
TABLE IV : Comparison of adaptation strategies under operational labeling constraints. Behavior-aware sampling achieves performance comparable to full-budget sampling while reducing the amount of labeled deployment traffic. Compared with confidence-based alternatives, it also provides feature-level interpretability.
Fig. 14 : Average drift indicator weights (a) show that the false positives and true positives cannot be effectively distinguished using the feature-drift indicator weights. The entropy trace (b) demonstrates a higher ambiguity in the indicators for false positives than true positives.
ϵ
TPR
Specificity
Balanced accuracy
0.0050
0.72±0.23
0.53±0.29
0.62±0.16
0.0075
0.74±0.23
0.52±0.28
0.63±0.16
0.0100
0.77±0.22
0.49±0.27
0.63±0.15
0.0125
0.79±0.21
0.48±0.26
0.63±0.14
0.0150
0.80±0.20
0.46±0.25
0.63±0.13
TABLE V : Detection results show low sensitivity to ϵ .
Fig. 15 : Temporal comparison of confidence-based and behavior-aware adaptation strategies.
Fig. 16 : Evaluation of the behavioral signal for adaptation. (a) Detector update strategies used to assess stability, consistency, and efficiency. (b) Corresponding evaluation metrics over the deployment period. Values close to one indicate that detector updates using only selected instances closely preserve the behavior of updates performed using all labeled deployment traffic.
The growth and heterogeneity of IoT devices create security challenges where static identification models can degrade as traffic evolves. This paper presents a two-stage, flow-feature-based pipeline for unsupervised IoT device traffic profiling and incremental model updating, evaluated on selected long-duration captures from the Deakin IoT dataset. For baseline profiling, density-based clustering (DBSCAN) isolates a substantial outlier portion of the data and produces the strongest alignment with ground-truth device labels among tested classical methods (NMI 0.78), outperforming centroid-based clustering on cluster purity. For incremental adaptation, we evaluate stream-oriented clustering approaches and find that BIRCH supports efficient updates (0.13 seconds per update) and forms comparatively coherent clusters for a held-out novel device (purity 0.87), but with limited capture of novel traffic (share 0.72) and a measurable trade-off in known-device accuracy after adaptation (0.71). Overall, the results highlight a practical trade-off between high-purity static profiling and the flexibility of incremental clustering for evolving IoT environments.
Sean M. Alderman, John D. Hastings
The Beacom College of Computer & Cyber Sciences, Dakota State University, Madison, SD, USA
Machine Learning as a Service (MLaaS) is a powerful cloud paradigm enabling data-driven intelligent applications in Internet of Things (IoT) environments, widely adopted across healthcare, smart homes, and industry due to its cost-effectiveness. However, the dynamic nature of IoT frequently alters data distributions, affecting MLaaS stability, while periodic MLaaS updates further introduce performance drift. Unlike traditional ML systems, MLaaS clients operate as black-box users without access to internal data or parameters, making drift detection particularly challenging. To address this, we propose a novel MLaaS Performance Drift Detection framework for IoT environments. The framework first employs an MLaaS extraction model that learns service behavior from input-output pairs and identifies prediction-influenced features. Building on this, the proposed MLaaS Performance Drift Detection (MPDD) model jointly captures variations in input data and MLaaS behavior. We further design an Adaptive-Temporal Performance Drift Detection Mechanism (APDDM) that dynamically adjusts monitoring frequency based on behavioral and data variations, enabling timely drift detection for effective service management. Extensive experiments on real-world datasets demonstrate that MPDD achieves up to 22-25% accuracy improvement over baseline drift detection methods. APDDM provides an average accuracy gain of approximately 4% and reduces the miss detection rate by around 9% compared to fixed-interval monitoring.
Deepak Kanneganti, Sajib Mistry, Sheik Mohammad Mostakim Fattah +3
School of Electrical Engineering, Computing and Mathematical Sciences (EECMS), Curtin University, Perth, WA, Australia · Department of Computing Science, Umeå University, Umeå, Sweden
Machine learning algorithms deployed for evolving streaming environments must handle the non-stationary data distributions, commonly referred to as concept drift. The presence of concept drift poses a major challenge for many real-world applications because it can severely degrade their predictive performance, hindering their ability to support robust decision-making. Consequently, the timely and efficient detection of drift events is critical for sustaining high accuracy over time. This study examines theoretically the concept drift characteristics and numerous drift detection algorithms across several categories. Furthermore, we evaluate their performance on both synthetic and real-world datasets exhibiting diverse streaming scenarios and drift characteristics, such as abrupt and gradual changes. This study aims to enhance understanding of the complex notion of concept drift characteristics and behavior of drift detectors, along with their applicability to diverse contexts.
Md Moman Ul Haque Khan, Samira Sadaoui
Department of Computer Science, University of Regina, Canada