Label-efficient Training Updates for Malware Detection over Time
Authors: Luca Minnei, Cristian Manca, Giorgio Piras, Angelo Sotgiu, Maura Pintor, Daniele Ghiani, Davide Maiorca, Giorgio Giacinto, Battista Biggio
Organizations: Department of Electrical and Electronic Engineering, University of Cagliari, Italy · Department of Computer, Control and Management Engineering, Sapienza University, Rome, Italy · CINI, Rome, Italy
Machine Learning (ML)-based detectors are becoming essential to counter the proliferation of malware. However, common ML algorithms are not designed to cope with the dynamic nature of real-world settings, where both legitimate and malicious software evolve. This distribution drift causes models trained under static assumptions to degrade over time unless they are continuously updated. Regularly retraining these models, however, is expensive, since labeling new acquired data requires costly manual analysis by security experts. To reduce labeling costs and address distribution drift in malware detection, prior work explored active learning (AL) and semi-supervised learning (SSL) techniques. Yet, existing studies (i) are tightly coupled to specific detector architectures and restricted to a specific malware domain, resulting in non-uniform comparisons; and (ii) lack a consistent methodology for analyzing the distribution drift, despite the critical sensitivity of the malware domain to temporal changes. In this work, we bridge this gap by proposing a model-agnostic framework that evaluates an extensive set of AL and SSL techniques, isolated and combined, for Android and Windows malware detection. We show that these techniques, when combined, can reduce manual annotation costs by up to 90% across both domains while achieving comparable detection performance to full-labeling retraining. We also introduce a methodology for feature-level drift analysis that measures feature stability over time, showing its correlation with the detector performance. Overall, our study provides a detailed understanding of how AL and SSL behave under distribution drift and how they can be successfully combined, offering practical insights for the design of effective detectors over time.
Figures & tables
Fig. 1: Overview of our proposed pipeline combining AL and SSL. The dash‑dot line represents the detector’s decision boundary. Given a set of newly collected unlabeled samples (left), AL queries a small set of informative samples for expert annotation (center), while SSL automatically assigns pseudo-labels to high-confidence samples (right), enabling efficient model retraining with reduced labeling cost.
Fig. 2: F1 score at FPR =1% over time for RF on ELSA (quarterly) and EMBER (monthly) with a 10% labeling budget. Top: comparison of AL strategies, alongside NR and FL references. Bottom: comparison of SSL strategies under the same budget, with the same references.
AL Strategy
1% Budget
2% Budget
5% Budget
10% Budget
ρ
F1
ρ
F1
ρ
F1
ρ
F1
SVM + NR
74.9
80.0
74.9
80.0
74.9
80.0
74.9
80.0
SVM + FL
82.2
85.4
82.2
85.4
82.2
85.4
82.2
85.4
SVM + RS
72.9
78.5
74.5
79.9
76.0
81.2
76.6
81.6
SVM + BADGE
78.6
82.9
80.2
84.0
81.5
85.0
83.5
86.3
SVM + CLUE
78.1
82.6
80.4
84.2
82.2
85.4
82.5
85.6
TABLE I: Average recall ( ρ ) and F1 score across different AL strategies and label budgets, at fixed FPR of 1%, for SVM and RF models on ELSA. Best overall results are in bold , the top AL method is underlined , and values below the NR baseline are in red .
AL Strategy
1% Budget
2% Budget
5% Budget
10% Budget
ρ
F1
ρ
F1
ρ
F1
ρ
F1
RF + NR
56.6
66.3
56.6
66.3
56.6
66.3
56.6
66.3
RF + FL
86.7
91.4
86.7
91.4
86.7
91.4
86.7
91.4
RF + RS
67.0
76.7
72.4
81.4
75.1
83.1
79.0
86.5
RF + Badge
69.0
78.3
73.5
81.9
79.6
86.5
84.8
90.3
RF + CLUE
61.7
71.3
67.6
76.8
74.9
83.1
80.7
87.6
TABLE II: Comparison of AL strategies for different label budgets on EMBER (RF). Refer to Table I caption for metrics and formatting.
SSL strategy
10% Budget
20% Budget
60% Budget
80% Budget
ρ
F1
ρ
F1
ρ
F1
ρ
F1
SVM + NR
74.9
80.0
74.9
80.0
74.9
80.0
74.9
80.0
SVM + FL
82.2
85.4
82.2
85.4
82.2
85.4
82.2
85.4
SVM + ST
73.8
79.1
73.7
79.0
73.5
78.9
73.9
79.3
SVM + AT
72.1
77.9
71.0
76.9
66.0
71.7
65.3
70.7
RF + NR
71.9
78.1
71.9
78.1
71.9
78.1
71.9
78.1
TABLE III: Average recall ( ρ ) and F1 score across different SSL strategies and pseudo-label budgets, at fixed FPR of 1%, for SVM and RF models on ELSA. Best overall results are in bold , the top SSL method is underlined , and values below the NR baseline are in red .
SSL strategy
10% Budget
20% Budget
60% Budget
80% Budget
ρ
F1
ρ
F1
ρ
F1
ρ
F1
RF + NR
56.6
66.3
56.6
66.3
56.6
66.3
56.6
66.3
RF + FL
86.7
91.4
86.7
91.4
86.7
91.4
86.7
91.4
RF + ST
52.1
60.3
52.3
60.9
48.1
55.7
48.1
56.1
RF + AT
50.8
58.5
51.6
59.8
47.4
54.6
48.3
56.0
TABLE IV: Comparison of SSL strategies for different pseudo-label budgets on EMBER (RF). Refer to Table III caption for metrics and formatting.
Fig. 3: F1 score at FPR =1% over time using RF on ELSA (quarterly) and EMBER (monthly), where each AL strategy is combined with its best-performing SSL strategy, including the optimal SSL budget, at a fixed labeling budget of 1% (ELSA) and 2% (EMBER). The lower panel reports the discriminant percentage of features over time, denoted by β , corresponding to the performance curves above.
Fig. 4: Scatter plots of F1 score versus β of the curve in Fig. 3 for ELSA (right plot) and EMBER (left plot). The insets report Pearson r and Kendall τ correlations with two-sided permutation-test p-values (10,000 resamples).
University of Technology Sydney Sydney, New South Wales, Australia · Yunnan University Kunming, Yunnan, China · University of New South Wales Sydney, New South Wales, Australia