Organizations: School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia · KDDI Research, Japan · School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia
Identification of IoT device types from passive traffic is increasingly used for security management in enterprise and ISP networks. However, the performance of machine learning-based classifiers gradually degrades under concept drift as device behavior evolves. Therefore, maintaining classification performance requires periodic retraining with newly labeled deployment traffic. The operational challenge is determining how much and which deployment traffic instances to label for maintaining classification performance. We show that these two decisions should be treated separately. While retraining solely on instances selected by a drift detector is prone to systematically overlooking parts of the emerging behavioral space, uniformly sampled deployment traffic captures more representative behavioral changes. Instead, drift detection is more effective at determining the amount of deployment traffic that should be labeled. We make three contributions. (1) We conduct a two-year longitudinal study of IoT traffic and characterize how behavioral evolution manifests across device classes and how retraining with newly labeled traffic restores classification performance. (2) We develop a conformity-based drift detector that captures class-conditional behavioral models directly from raw traffic features and provides feature-level explanations of behavioral evolution. (3) We demonstrate that adjusting the traffic labeling rate according to the observed behavioral evolution, combined with uniform traffic sampling, maintains classifier performance more effectively than detector-guided sample selection and is beneficial to managing the traffic labeling effort. We further show that this strategy performs comparably to confidence-guided adaptation while providing feature-level explanations. Our evaluation uses 3.8 million IPFIX flow records collected from 21 IoT types over more than 2 years.
Figures & tables
Device Make-and-Model
Category
Amazon Echo Show 5
Voice assistant
Amazon Echo Gen 2
Voice assistant
Google Nest Mini
Voice assistant
Amazon Echo Dot with Clock
Voice assistant
Google Home
Voice assistant
Sony Smart Speaker
Voice assistant
TABLE I : The 21 IoT devices studied.
Feature Name
Description
Abbr. Name (reverse)
packetTotalCount
# of packets.
pktTotCnt (rPktTotCnt)
octetTotalCount
# of bytes (header and payload).
octTotCnt (rOctTotCnt)
smallPacketCount
# of packets with less than 60 Bytes
of payload.
sPktCnt (rSPktCnt)
largePacketCount
# of packets with at least 220 Bytes
of payload.
lPktCnt (rLPktCnt)
TABLE II : Traffic features in IPFIX flow records.
Fig. 1 : Behavioral evolution during long-term deployment progressively degrades classifier performance. The partial restoration observed in late 2023 suggests that device behaviors may evolve non-monotonically away from the training distribution.
Fig. 2 : Behavioral evolution manifests differently across IoT device classes. Representative examples show persistent behavior (a), behavioral decay (b), and behavioral restoration (c), while the corresponding t-SNE projection (d) illustrates how changes in traffic distribution explain these different evolution patterns. The black curves show the upper-bound maintenance performance obtained by retraining with all newly labeled deployment traffic.
Fig. 3 : Upper-bound maintenance performance established by retraining the classifier using exhaustively labeled deployment traffic. This experiment demonstrates that long-term classifier maintenance is feasible, but highlights the prohibitive labeling effort required in practice.
Fig. 4 : Feature-level view of a correctly classified deployment window for Philips Hue Bridge. Although the observed feature values (red triangles) occupy only a subset of the training observations (gray markers), they remain within previously learned behavioral patterns, illustrating behavioral conformity despite distributional differences.
Fig. 5 : Drifted instances reveal feature-level non-conformity with respect to the predicted class. Even when both flows are misclassified as Apple HomePod, different features expose their deviation from the training distribution.
Fig. 6 : Modeling the behavioral range of non-negative traffic features. (a) GKDE assigns non-zero probability to infeasible feature values ( x<0 ), which is invalid for non-negative features. (b) Log-normal KDE (LNKDE) preserves non-negativity while accurately modeling the observed behavioral range of the feature.
Fig. 7 : Construction of feature-level conformity regions. The conformity threshold θ is chosen so that the probability mass of the non-conforming region (shaded) does not exceed the user-defined sensitivity parameter ϵ .
Fig. 8 : Relationship between behavioral non-conformity and classifier misclassification. Mean TPR (a) and specificity (b) are reported for each device class over the deployment phase. Error bars indicate temporal standard deviation across evaluation windows.
Fig. 9 : Temporal evolution of detector performance averaged across device classes. TPR remains relatively stable throughout deployment, whereas specificity gradually decreases as behavioral evolution introduces an increasing number of correctly classified instances exhibiting previously unseen patterns.
ϵ
TPR
Specificity
Balanced accuracy
0.001
0.65±0.27
0.58±0.31
0.61±0.17
0.010
0.77±0.22
0.49±0.27
0.63±0.15
0.100
0.95±0.08
0.26±0.19
0.60±0.10
TABLE III : Detection results for different ϵ values.
Fig. 10 : Feature-level explanations of behavioral evolution for the Wansview Q5 IP camera. Misclassifications across two deployment periods are explained by distinct subsets of non-conforming traffic features.
Fig. 11 : (a) Classification performance under different adaptation strategies. Detector-guided adaptation (solid green) substantially improves performance over no adaptation (dotted blue), but remains consistently below the upper-bound maintenance strategy (dashed gray). Replacing detector-selected retraining instances with uniformly sampled deployment traffic at the same detector-guided labeling rate (dash-dotted green) yields performance close to the upper bound. (b) Fraction of deployment traffic labeled in each evaluation window. Detector-guided adaptation labels progressively fewer instances over time as fewer deployment flows are identified as behaviorally non-conforming, whereas uniform sampling preserves the detector-guided sampling rate while providing more representative coverage of the evolving behavioral patterns. For the strategy involving random sampling, error bars indicate the standard deviation across 10 random seeds.
Fig. 12 : Detector-selected instances (red crosses) concentrate around only a subset of the emerging behavioral patterns (orange circles). Uniformly sampled instances (black stars) provide substantially broader coverage of the evolving traffic behavior, explaining their superior adaptation performance.
Fig. 13 : Classifier maintenance under a fixed operational labeling budget. Sampling uniformly at the full budget (dash-dotted pink) provides a reference operating point. Reducing the sampling rate uniformly across deployment (dotted orange) noticeably degrades adaptation performance. In contrast, adjusting the sampling rate according to the observed behavioral evolution (solid maroon) achieves comparable performance while labeling fewer deployment instances. Error bars are omitted for visual clarity.
Metric
Cap. rate
Uniform below cap.
Behavior-aware rate
Low-confidence
Low-confidence stratified
Confidence-guided rate
Avg. macro F1-score
0.8970±0.0042
0.8797±0.0067
0.8934±0.0049
0.8845
0.8905
0.8942±0.0039
Labeling cost
0.1%(Cap)
0.0501±0.0022%
0.0826±0.0006%
0.1%(Cap)
0.123%
0.0867±0.0009%
TABLE IV : Comparison of adaptation strategies under operational labeling constraints. Behavior-aware sampling achieves performance comparable to full-budget sampling while reducing the amount of labeled deployment traffic. Compared with confidence-based alternatives, it also provides feature-level interpretability.
Fig. 14 : Average drift indicator weights (a) show that the false positives and true positives cannot be effectively distinguished using the feature-drift indicator weights. The entropy trace (b) demonstrates a higher ambiguity in the indicators for false positives than true positives.
ϵ
TPR
Specificity
Balanced accuracy
0.0050
0.72±0.23
0.53±0.29
0.62±0.16
0.0075
0.74±0.23
0.52±0.28
0.63±0.16
0.0100
0.77±0.22
0.49±0.27
0.63±0.15
0.0125
0.79±0.21
0.48±0.26
0.63±0.14
0.0150
0.80±0.20
0.46±0.25
0.63±0.13
TABLE V : Detection results show low sensitivity to ϵ .
Fig. 15 : Temporal comparison of confidence-based and behavior-aware adaptation strategies.
Fig. 16 : Evaluation of the behavioral signal for adaptation. (a) Detector update strategies used to assess stability, consistency, and efficiency. (b) Corresponding evaluation metrics over the deployment period. Values close to one indicate that detector updates using only selected instances closely preserve the behavior of updates performed using all labeled deployment traffic.
School of Electrical Engineering, Computing and Mathematical Sciences (EECMS), Curtin University, Perth, WA, Australia · Department of Computing Science, Umeå University, Umeå, Sweden