What Malware Actually Does: An Empirical Analysis of Post-Infection Behavior Across Malware Families
Authors: Emmanuela Andam, Emanuel S. Grant, Naima Kaabouch
Organizations: Artificial Intelligence Research (AIR) Center, University of North Dakota, Grand Forks, ND, USA · School of Electrical Engineering and Computer Science, University of North Dakota, Grand Forks, ND, USA
Most malware detection research focuses on identifying whether a file is malicious before it executes. Far less attention goes to what happens after infection. This paper presents an empirical analysis of post-infection malware behavior across 37 families, grounded in dynamic sandbox execution using CAPEv2. Behavioral traces are modeled as heterogeneous graphs with five node types and seven typed edge relationships, preserving relational structure that flat feature vectors discard. A voting ensemble trained on graph-derived structural features achieves 94.44% accuracy and MCC of 0.863, outperforming a flat behavioral baseline by 7.4 percentage points in MCC. A multiclass classifier distinguishes 24 malware families, and a per-family behavioral fingerprint analysis characterizes post-infection activity across registry, network, process, and API dimensions. An LLM explanation layer generates analyst-readable reports with MITRE ATT&CK mappings grounded in observed behavioral evidence. The full pipeline is deployed as HeteroScan at https://ama-andam.github.io/heteroscan.
Figures & tables
Fig. 1: Existing detection approaches classify executables as malicious or benign at the perimeter but do not analyze post-infection behavior. This work addresses the gap by empirically characterizing what malware actively does inside a compromised system.
Fig. 2: HeteroScan pipeline: Windows executables (malicious and benign) are submitted to the CAPE sandbox, behavioral reports are parsed into heterogeneous graphs, structural features are extracted for ensemble classification, and an LLM generates analyst-readable behavioral reports.
Fig. 3: Dataset overview: top 20 malware families by sample count (left) and overall dataset composition (right). The full dataset comprises 8,047 samples across 37 malware families and benign executables from four sources.
Fig. 4: Heterogeneous behavioral graph structure. Five node types and seven typed edge relationships capture the post-infection system entity interactions observed during sandbox execution.
Category
Count
Description
Raw node counts
5
Count per node type
Total node count
1
Sum of all node counts
Pairwise ratios
7
API/proc, reg/proc, net/proc, etc.
Log transforms
5
log(1+n) per node type
Node statistics
30
Mean, std, max, min, sum, active
fraction per node type (6 × 5)
TABLE IV: Feature Categories (55 Total)
Classifier
Key Hyperparameters
Random Forest
n_estimators=500, max_features=sqrt,
class_weight=balanced
Extra Trees
n_estimators=500, max_features=sqrt,
class_weight=balanced
XGBoost
n_estimators=500, max_depth=6,
learning_rate=0.05, subsample=0.8,
TABLE V: Classifier Hyperparameters
Fig. 5: Model performance comparison across five metrics on the test set (left) and confusion matrix for the voting ensemble (right). The dashed red line indicates the 90% accuracy threshold.
Model
Acc
F1
Prec
Rec
MCC
Random Forest
0.9471
0.9638
0.9399
0.9888
0.8698
Extra Trees
0.9418
0.9600
0.9395
0.9814
0.8561
XGBoost
0.9497
0.9647
0.9630
0.9665
0.8772
Ensemble
0.9444
0.9617
0.9429
0.9814
0.8628
TABLE VI: Classification Results on Test Set (378 samples: 269 malicious, 109 benign)
Representation
Features
Acc
F1
MCC
Flat behavioral counts
5
0.9127
0.9399
0.7828
Graph-derived features
55
0.9418
0.9596
0.8565
Improvement
+50
+0.029
+0.020
+0.074
TABLE VII: Graph-Derived Features vs. Flat Behavioral Counts
Fig. 6: Top 20 most important features ranked by mean decrease impurity from the Random Forest classifier. Edge count features, particularly e_calls and e_reads_reg , dominate the ranking.
Model
Acc
F1
MCC
FP/FN
Heterogeneous GNN †
0.763
0.846
0.382
141 / 30
Ensemble
0.944
0.962
0.863
16 / 5
TABLE VIII: Ensemble vs. GNN Baseline
Metric
Value
Accuracy
0.6891
Macro F1
0.5818
Weighted F1
0.6773
Families
24
Test samples
267
TABLE IX: Multiclass Family Classification Results
Family
Proc
File
Reg
Net
API
BlackMatter
5
1
5036
1
118
QuasarRAT
5
1
1190
2
147
WSHRAT
3
1
1136
2
98
DCRat
5
1
1025
1
137
NanoCore
5
1
734
3
143
Babadeda
5
1
814
2
102
TABLE X: Median Behavioral Statistics Per Family
Fig. 7: HeteroScan analysis output for a NanoCore sample showing the MALICIOUS verdict with 99.8% ensemble confidence, behavioral graph visualization, graph statistics, and node distribution across process, API, file, registry, and network entity types.
Family
Proc
API
File
Reg
Net
Conf
njRAT
3
133
1
461
3
99.9%
AsyncRAT
4
117
1
1,264
1
97.3%
NanoCore
4
122
1
218
1
99.8%
TABLE XI: Behavioral Graph Statistics for Case Study Samples
Traditionally, machine learning methods for PE malware detection have relied on static features like byte histograms, string information, and PE header contents. One barrier to incorporating dynamic analysis features has been the semi-structured nature of sandbox behavior reports. We show that, using the latest generation of large language models with reasoning, it is possible to efficiently process these behavior reports and utilize them as part of a malware detection pipeline. Specifically, we leverage LLMs to generate behavior-based malware detection rules based on a small training set of labeled malware. We find that these detection rules, derived from behavioral features, are much more robust to concept drift than standard static-feature methods, while maintaining practical false positive rates. Finally, we introduce Trident, a system which combines a classic decision tree model over static features, our behavior-based detection rules, and direct LLM analysis of sandbox reports through majority voting. Trident outperforms standard methods using static features, outperforms behavior-based rules alone, and is as resilient to concept drift as active learning methods without requiring retraining.
Rebecca Saul, Jingzhi Jiang, Elliott Chia +1
University of California, Berkeley Berkeley, California, USA
Malware detection remains largely reactive: machine learning models trained on known samples degrade as threats evolve. Understanding evolutionary relationships among malware families can inform proactive defense, but traditional reverse engineering can take months to years to uncover such lineage relationships. We propose MalTree, a framework that applies bioinformatics inspired phylogenetic techniques (UPGMA and Neighbor-Joining) at scale to model malware evolution automatically using structural, behavioral, and image-based features. We introduce temporal validation using VirusTotal timestamps to assess whether inferred trees reflect actual evolutionary order. MalTree achieves 87% temporal consistency, indicating that inferred evolutionary relationships closely align with real-world emergence timelines. Our analysis shows that some families mutate over 10 times faster than others, suggesting that detection strategies should be tailored to family-specific evolutionary tempos. Case studies, including the Mirai botnet, confirm that inferred relationships from our phylogenetic tree align with documented threat intelligence. Our framework provides a foundation for shifting malware analysis from sample-by-sample classification toward lineage-aware evolutionary modeling.
Akash Amalan, Georgios Smaragdakis, Tom J. Viering
Delft University of Technology, Delft, The Netherlands.
Malware family classification remains a challenging task in automated malware analysis, particularly in real-world settings characterized by obfuscation, packing, and rapidly evolving threats. Existing machine learning and deep learning approaches typically depend on labeled datasets, handcrafted features, supervised training, or dynamic analysis, which limits their scalability and effectiveness in open-world scenarios. This paper presents a zero-label malware family classification framework based on a weighted hierarchical ensemble of pretrained large language models (LLMs). Rather than relying on feature-level learning or model retraining, the proposed approach aggregates decision-level predictions from multiple LLMs with complementary reasoning strengths. Model outputs are weighted using empirically derived macro-F1 scores and organized hierarchically, first resolving coarse-grained malicious behavior before assigning fine-grained malware families. This structure enhances robustness, reduces individual model instability, and aligns with analyst-style reasoning.