Classifier-Dependent Benefits of Pseudo-Labeling for Semi-Supervised Android Malware Attribution

Md Rafid Islam; Zahid Hasan; Hafiz Abdur Rahman

cs.LGAug 26, 2026

Classifier-Dependent Benefits of Pseudo-Labeling for Semi-Supervised Android Malware Attribution

Authors: Md Rafid Islam, Zahid Hasan, Hafiz Abdur Rahman

Organizations: Department of Electrical and Computer Engineering North South University Dhaka, Bangladesh

Abstract

Detecting and classifying Android malware families remains challenging due to high feature dimensionality, class imbalance, and the high cost of expert-labeled data. Semi-supervised learning (SSL) offers a way to leverage unlabeled samples, but prior works rarely test whether SSL benefits generalize across classifier types or report statistical significance. We present a systematic evaluation of pseudo-labeling across six classifiers (LightGBM, XGBoost, Random Forest, Logistic Regression, MLP, and SVM) on the CICMalDroid 2020 dataset, using five-fold stratified cross-validation and paired t-tests across five labeled ratios (1-20%). We find that SSL benefit is strongly classifier-dependent: SVM shows the largest significant gain (+4.4% accuracy at 5% labels, p = 0.0028), LightGBM improves modestly (+0.8 to +1.3% at 2-5% labels), while Random Forest is significantly harmed at low label ratios (-3.1% at 1% labels). Per-class analysis reveals SSL disproportionately benefits the hardest-to-classify families, with Adware F1 improving by +13.8 percentage points versus only +0.8 for the already well-classified Benign class. We further show that approximately 800 labeled samples (10% of the dataset) yield near-optimal performance across all classifiers. These findings offer practical guidance on when and with which classifier pseudo-labeling is worthwhile for Android malware classification.

Figures & tables

Category	Samples	Description
Adware	1253	Displays intrusive ads or collects user data for advertising.
Banking	2100	Targets banking credentials or financial transactions.
SMS Malware	3904	Abuses SMS APIs to send messages or intercept codes.
Riskware	2546	Potentially harmful apps that pose security/privacy risks.
Benign	1795	Legitimate apps with no malicious behavior.

TABLE I: Class-wise distribution and descriptions in the CICMalDroid 2020 dataset.

Classifier	Library	Key hyperparameters
LightGBM (LGBM)	lightgbm	leaves=31, lr=0.05
XGBoost (XGB)	xgboost	estimators=200, lr=0.05
Random Forest (RF)	scikit-learn	estimators=100
Logistic Regression (LR)	scikit-learn	max_iter=1000
MLP	scikit-learn	layers=(100,50), early stop
SVM	scikit-learn	RBF kernel, prob.=True

TABLE II: Classifier configurations.

Fig. 1: Summary of the experimental methodology.

Classifier	Test Accuracy	Test Macro F1
LightGBM (LGBM)	94.66%	93.61%
XGBoost (XGB)	94.34%	93.31%
Random Forest (RF)	94.08%	92.87%
MLP	90.29%	88.39%
Logistic Regression (LR)	84.77%	82.55%
SVM	77.93%	74.38%

TABLE III: Fully supervised upper bound performance (100% labels).

Classifier	1%	2%	5%	10%	20%
		Labels	Labels	Labels	Labels
LGBM	$+0.7$	$\mathbf{+1.3^{*}}$	$\mathbf{+0.8^{*}}$	$+0.3$	$-0.1$
	$\pm 1.8$	$\pm 1.3$	$\pm 1.3$	$\pm 1.0$	$\pm 0.9$
XGB	$+0.3$	$0.0$	$+0.3$	$+0.5$	$+0.3$
	$\pm 1.7$	$\pm 1.9$	$\pm 1.2$	$\pm 1.0$	$\pm 0.7$
RF	$\mathbf{-3.1^{\dagger}}$	$\mathbf{-1.6^{\dagger}}$	$-0.9$	$-0.5$	$+0.1$

TABLE IV: SSL improvement over supervised baseline (gap in percentage points).

Class	Supervised F1 (%)	SSL F1 (%)	Change (pp)
Adware	$12.5\pm 7.5$	$26.3\pm 8.4$	$\mathbf{+13.8}$
Banking	$36.5\pm 7.5$	$44.3\pm 5.8$	$\mathbf{+7.8}$
SMS Malware	$70.7\pm 3.6$	$76.3\pm 2.1$	$\mathbf{+5.6}$
Riskware	$53.7\pm 4.6$	$57.7\pm 3.7$	$\mathbf{+3.9}$
Benign	$62.8\pm 2.9$	$63.6\pm 2.1$	$+0.8$

TABLE V: Per-class F1 scores for SVM at 5% labels.

Fig. 2: Accuracy vs. labeled data ratio for all six classifiers. Solid blue lines represent supervised learning; dashed orange lines represent SSL with pseudo-labeling. Shaded regions indicate $\pm 1$ standard deviation across five-fold cross-validation. SSL shows clear benefit for SVM at 5% labels and modest benefit for LGBM at 2–5% labels, while degrading RF at low label ratios.

Classifier	1%	2%	5%	10%	20%
		Labels	Labels	Labels	Labels
LGBM	23.3	14.6	9.1	5.8	2.8
XGB	25.1	15.2	9.5	5.9	2.7
RF	27.2	18.9	11.1	6.8	3.5
LR	20.1	14.4	9.3	5.6	2.7
MLP	44.1	36.6	18.6	11.0	6.5

TABLE VI: Gap to upper bound under SSL (percentage points).

Pseudo-Labeling

Pseudo-labeling uses a model’s predictions as training targets for unlabeled data, reducing dependence on costly human annotations.

See all 156 papers

SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification
Jun 30, 2026Rui Shu, Tianpei Xia, Jingzhu HeSelf-Training Cybersecurity
North Carolina State University · ShanghaiTech University
Trust Under Siege: Label Spoofing Attacks against Machine Learning for Android Malware Detection
Mar 14, 2025Tianwei Lan, Luca Demetrio, Farid Nait-Abdesselam +2Android Malware Detection Malware Classification
1Université Paris Cité, France · University of Genova, Italy · 3INRIA, France +1
Self-Supervised Learning for Android Malware Detection on a Time-Stamped Dataset
Apr 24, 2026Annan Fu, Hao Pei, Maryam TanhaConcept Drift Android Malware Detection
Khoury College of Computer Sciences Northeastern University Vancouver, Canada

Jun 30, 2026J/K move · Enter open · S save

SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification

Rui Shu, Tianpei Xia, Jingzhu He

North Carolina State University · ShanghaiTech University

Self-Training Cybersecurity Binary Classification Semi-Supervised Learning Hyperparameter Optimization

Open paper AskPDF arXiv

Background. Labeled data for security classification is scarce. Semi-supervised learning (SSL) propagates labels from a small labeled pool to larger unlabeled pools. Yet security applications often use SSL as a black box: default parameters, a fixed classifier, and no handling of pseudo-label-induced class imbalance. Aims. Recent work reports sizeable gains from optimizing SSL pipelines via joint search, AutoML, or per-component tuning. These gains are hard to attribute: they may reflect useful SSL-classifier interactions, or mostly from simply tuning the downstream classifier. We disentangle these effects for binary tabular security data with classical SSL and tree-based classifiers. Method. We build SemiScope as an analysis instrument, not a deployment recommendation. It uses Bayesian Optimization to jointly tune SSL settings, confidence filtering, oversampling, and the classifier. The key control, Tuned-Clf, fixes SSL to defaults but gets the same 100-trial classifier budget and validation-set threshold tuning as SemiScope. At 10% labels, we compare them with paired TOST using a +/-1.0 g-measure smallest effect of interest. Results. SemiScope beats every default SSL baseline on all five datasets, improving over the strongest by 0.7-12.7 points. Under the equal-budget control, Tuned-Clf is statistically equivalent to the full pipeline on 4 of 5 datasets; Phishing is inconclusive. Classifier HPO alone recovers a median 86% of SemiScope's gain over Default Self-Training (ST) + Random Forest (RF). Conclusions. The reusable contribution is the decomposition protocol. A simpler recipe suffices: use Self-Training, tune the classifier with Bayesian Optimization, and tune the decision threshold on validation data. It reaches within 1 g-measure of Supervised RF at 20-30% labels on four datasets and 40% on Drebin, at the same or lower label rate than Default ST + RF on every dataset.

Abstract

Pseudo-Labeling

Explore similar work

SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification

Trust Under Siege: Label Spoofing Attacks against Machine Learning for Android Malware Detection

Self-Supervised Learning for Android Malware Detection on a Time-Stamped Dataset