Malware Clustering in the Wild: Taxonomy, Challenges, and Evaluation Metrics
Organizations: University of Maryland, Baltimore County, Department of Computer Science and Electrical Engineering, Baltimore, 21250, Maryland, USA
Abstract
As malware continues to grow in volume, diversity, and sophistication, clustering has emerged as a critical technique for analyzing large malware corpora. Clustering enables the discovery of structural relationships, behavioral patterns, and evolutionary trends, and is widely used to group unknown samples, infer malware family boundaries, and identify novel variants. However, existing research remains fragmented, employing a wide range of clustering techniques with no unified perspective that systematically organizes prior work or clearly identifies gaps in the literature. This survey addresses this gap by organizing previous malware clustering research into seven categories: centroid-based, hierarchical-based, density-based, distribution-based, mixed- type, fuzzy, and AI-driven approaches. In addition, this survey summarizes publicly available malware datasets commonly used in clustering research and examines the feature representations derived from these datasets. Our goal is to clarify the current landscape and outline promising avenues for future research on malware clustering, including how AI-driven methods may shape large-scale and reproducible analysis.
Figures & tables
| Paper | Year | Feature | Dataset | Size | Model | Metrics |
| [ 12 ] | 2021 | Byte bigrams (Top-20, norm.) | Malicia (20 fam., balanced) | 20k | K-Means | ARI |
| [ 6 ] | 2021 | Engineered statics (67) | Drive-by-download | 2k | K-Means++ (imwk-means) | Euclidean |
| [ 71 ] | 2021 | Static + dynamic (Cuckoo) | 18 fam. (imbal.) | not reported | BIRCH (online) | CF-tree radius |
| [ 95 ] | 2022 | Strings vs IAT | Windows ransomware | not reported | Similarity clustering | Jaccard |
| [ 59 ] | 2022 | Android statistics | Android malware (RT) | not reported | K-Means (auto- ) | not reported |
| [ 28 ] | 2022 | Android APK statics | Android apps | 10.7k | Mahalanobis clustering | Mahalanobis |
| Paper | Year | Feature | Dataset | Size | Model | Metrics |
| [ 83 ] | 2021 | Encoded network traffic features | CIC-IDS2017, FCIC-DDoS19 | 1.08M | GMM | Confusion matrix, Ac, Pr, Re, and F1 |
| [ 23 ] | 2021 | Log-transformed network traffic features | CIC-IDS2017 | 1.08M | GMM | Pr, Re, and F1 |
| [ 37 ] | 2022 | 3D vectors compressed by autoencoders | IoT Network Intrusion Dataset | 2.99M | VBGMM, Kmeans, GMM | Purity, Rand Index, F-score, and ARI |
| [ 7 ] | 2022 | Integrated features from multi autoencoders | KDDcup99, CICMalDroid20, AWID2018 | 15K, 80K, 60K | GMM | F1 |
| [ 1 ] | 2023 | Permission and API based features | TUNADROMD and others | 4465 | GMM, Kmeans, others | Ac |
| [ 84 ] | 2023 | Vectors of malware static memory features | CICMalMem22 Malware-Exploratory | 58K 216K | GMM Kmeans DBSCAN | Consistency across clustering graphs |
| Paper | Year | Feature | Dataset | Size | Model | Metrics |
| [ 11 ] | 2020 | TLSH hash | an IoT dataset | 12993 | k-medoid OPTICS updated OPTICS | Purity |
| [ 26 ] | 2021 | FastText and LSTM encoded SCDGs | Proposed Win PE dataset | 6615 | Proposed model DBSCAN, OPTICS Ward Agglo [ 87 ] | SI, ARI FMI V-measure |
| [ 66 ] | 2022 | Static information behavior network traffic | MalDroid2020 [ 20 ] | 11598 | KMeans, KNN AGNES, DIANA [ 46 ] DBSCAN, OPTICS Spectral [ 91 ] | RI, MIBS FMS, SC CH |
| [ 60 ] | 2023 | IoT network traffic | N-BaIoT [ 56 ] | 7M | DBSCAN-GWO | Ac, Pr, Re, DR |
| [ 52 ] | 2023 | Enhanced ACFG | BODMAS [ 96 ] | 57K | DBSCAN, KNN OPTICS | Ac, Coverage Macro Pr Macro Re |
| Paper | Year | Feature | Dataset | Size | Model | Metrics |
| [ 62 ] | 2024 | 56 features Static | CIC-MalMem-2022 | 58,596 X 56 | LR+SVM +KNN+NB | Ac, FPR, and DR |
| [ 15 ] | 2024 | DLL imports and PE features | A subset of SOREL-20M | 1.1M | HDBSCAN | PPV, TPR, TNR, FPR, FNR, NPV |
| [ 78 ] | 2025 | Dissemble Opcodes | A subset of SOREL-20M | 159 | GMM, Kmeans Agglo | ARI, AMII, and FMI |
| [ 65 ] | 2024 | runtime signals, network traffic, encryption behaviors | private | n/r | HTSP | Ac, precision, recall, F1-score |
| [ 31 ] | 2024 | Opcodes | -(random) | 284 | hierarchical-KNN | Ac, precision, recall |
| [ 47 ] | 2025 | Opcodes | KISA | 159 | CNN | Ac, precision, recall, F1-score |
| Paper | Year | Feature | Dataset | Size | Model | Metrics |
| [ 34 ] | 2009 | Not specified | Kingsoft AVLab | 3198 samples (3 datasets) | Hierarchical K-means | F1 score |
| [ 99 ] | 2010 | Function based instruction | Kingsoft AVLab | No Infor. | Hierarchical K-mediods | F1 score |
| [ 4 ] | 2020 | Not specified | VirusTotal | No Infor. | K-Means Hierarchical | Purity score |
| [ 94 ] | 2021 | Static features | CICAndMal2017 | 7067 | New model | AUC, Pr, Re, and F1 |
| [ 88 ] | 2021 | App permissions | Drebin | 129,000 | Fuzzy clustering LightGBM | Pr, Re |
| [ 27 ] | 2023 | Not specified | Kaggle EMBER2018 BODMAS | 216k 134k 1M | K-Means multi classifiers | Ac F1 score |
| Dataset Name | Year | OS | Features | Dataset Size |
| SOREL-20M [ 35 ] | 2021 | Windows | PE | 20M |
| Ransmap [ 36 ] | 2025 | Windows | Storage and Memory access patterns | NI |
| AndroOBFS [ 49 ] | 2022 | Android | Time tags, family information | 16279 |
| BODMAS [ 97 ] | 2021 | Windows | PE | NI |
| EMBER2024 [ 43 ] | 2025 | Multi-sys | Previously undetected samples | 3.2M |
| MOTIF [ 42 ] | 2023 | Windows | Malware tagged with families | 3095 |
| Feature | Centroid | Hierarchical | Distribution | Density |
| Bytes | 1 | - | - | - |
| Strings | 1 | - | - | - |
| IAT | 1 | - | - | - |
| API Calls | - | 1 | 1 | - |
| AV Labels | - | - | - | - |
| Registry Data | 1 | - | - | - |
Clustering
Clustering groups data points by shared structure, developing methods to discover, validate, and interpret groups across diverse data and constraints.
See all 370 papers