Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution under Analysis Budgets

Mubashar Iqbal; Asifullah Khan; Hifsa Asif; Saddam Hussain Khan; Umme Zahoora; Asifullah Khan

cs.CRSep 4, 2026

Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution under Analysis Budgets

Authors: Mubashar Iqbal, Asifullah Khan, Hifsa Asif, Saddam Hussain Khan, Umme Zahoora, Asifullah Khan

Organizations: Pattern Recognition Lab, DCIS, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, Pakistan · Deep Learning Lab, Center for Mathematical Sciences, PIEAS, Nilore, Islamabad, Pakistan · Interdisciplinary Research Center for Smart Mobility and Logistics, King Fahd University of Petroleum & Minerals, Dhahran 31261, Saudi Arabia · Department of Creative Technology, AIR University, E9 Campus, Islamabad, Pakistan. ·

Abstract

Sandbox execution and memory forensics are among the most constrained resources in malware triage. Static analysis can scale to millions of files, whereas dynamic and memory analysis require minutes of analyst controlled infrastructure for each sample. Despite this difference, multimodal ransomware detectors often apply every modality to every sample, causing analysis cost and time to verdict to increase linearly with sample volume even when static evidence is already sufficient for a decision. We present a cost aware Hierarchical Multi-Agent System that formulates evidence acquisition as a budgeted sequential decision problem. Specialist agents generate schema validated risk signals for each modality, domain controllers aggregate these signals, and a Meta-Orchestrator begins with static evidence and escalates to dynamic and memory evidence only when confidence is insufficient or agents within a controller disagree. An optional, bounded, locally hosted large language model reviewer can adjust a verdict by at most one tier but cannot replace the deterministic pipeline. Each decision is recorded with a complete provenance trace. In multiple runs over 12439 samples from 16 ransomware families and benign samples, the deterministic HMAS achieves F1 0.93 and macro F1 0.97, resolving 57.95% of cases using static evidence alone, 35.83% after adding dynamic evidence, and only 6.21% through the full pipeline. The average internal analysis cost is 6.65 units, compared with 12 for exhaustive analysis, representing a 44.6% reduction. Standalone leave-one-family-out testing further shows that accuracy on families held out during tuning falls to 0.26 to 0.64 outside the Benign and high support classes. We report these results alongside a cost sensitivity analysis, a partial leave one component out ablation, and a full scale comparison with learned early and late fusion and cascade baselines.

Figures & tables

Method	Modalities	Budget-aware	Family attrib.	Memory evidence	Audit trace
XRan [ 12 ]	Dynamic	No	No	No	Explanations
RansoGuard [ 14 ]	Dynamic (early)	Partially (early stop)	No	No	No
Pulse [ 15 ]	Static (assembly)	No	No	No	No
RansomFormer [ 7 ]	Static + dynamic	No	No	No	No
volGPT [ 6 ]	Memory	No	Process-level	Yes	LLM text
Iqbal & Khan [ 19 ]	Static + dynamic + network	No	Yes	No	Agent trace

Table 1: Positioning of HMAS. “Budget-aware” means the method decides at inference time which evidence to acquire.

Figure 1: HMAS architecture: feature-extraction pipeline (left), hierarchical decision pipeline (right), and the optional LLM reasoning layer.

Figure 2: HMAS architecture: feature-extraction pipeline (left), hierarchical decision pipeline (right), and the optional LLM reasoning layer.

Figure 3: Runtime pipeline: static evidence is processed first; the Meta-Orchestrator escalates on insufficient confidence or disagreement, fuses the acquired scores, and optionally invokes the bounded reviewer.

Figure 4: HMAS implementation stack: evidence generation (CAPEv2 sandbox with 24 guest VMs), the asynchronous Python runtime with Pydantic-typed messages and scikit-learn, XGBoost, LightGBM and PyTorch specialist models, and the AutoGen [ 31 ] reviewer on a local Ollama-hosted model (Table 2 ).

Item	Setting
Backend	Ollama + AutoGen ( AssistantAgent , RoundRobinGroupChat )
Models	Phi-3 Mini; Qwen2.5-3B
Context / output	4096 / 512 tokens
Sampling	$T{=}0.1$ , top- $p{=}0.85$ , top- $k{=}20$ , repetition penalty 1.05, seed 42
Input / output	Structured agent evidence / one of three actions

Table 2: Reviewer LLM configuration.

Figure 5: Class distribution (16 ransomware families and benign).

Policy	Acc.	Prec.	Rec.	F1	ROC-AUC	Avg. modalities
B1 Static-only	0.9130	0.9988	0.8273	0.9050	0.9983	1.00
B2 Dynamic-only	0.9329	0.9642	0.8994	0.9307	0.9883	1.00
B3 Memory-only	0.5002	1.0000	0.0005	0.0010	0.7644	1.00
B6 Static+Dynamic (fixed)	0.9647	0.9993	0.9302	0.9635	0.9995	2.00
B7 Exhaustive (fixed)	0.9401	0.9993	0.8811	0.9364	0.9995	3.00
H0 HMAS (deterministic)	0.9412	0.9993	0.8831	0.9376	0.9989	0.618

Table 3: Binary detection, full-scale policies, default 0.5 threshold, single run (seed 42).

Policy ( $n=100$ )	Acc.	Prec.	Rec.	F1	ROC-AUC	PR-AUC (AP)
B4 Early fusion (learned)	0.93	0.93	1.00	0.9637	0.96	1.00
B5 Late fusion (stacked)	0.94	0.9394	1.00	0.9688	1.00	1.00
B8 Fixed cascade	0.99	1.00	0.9892	0.9945	1.00	1.00
B9 Learned budgeted gate	0.98	0.9892	0.9892	0.9892	1.00	1.00

Table 4: Binary detection

Figure 6: F1 vs. mean internal cost for the four policies with a directly logged decision latency value

Reference	Compared policy	Ref.-only correct	Policy-only correct	$p$ -value
B1 Static-only	B6 Static+Dynamic	207	850	$1.1\times 10^{-92}$
B1 Static-only	B7 Exhaustive	412	749	$6.1\times 10^{-23}$
B1 Static-only	H0 HMAS	342	1054	$3.0\times 10^{-84}$
B7 Exhaustive	B6 Static+Dynamic	0	306	$3.1\times 10^{-92}$
B7 Exhaustive	H0 HMAS	344	719	$8.8\times 10^{-31}$

Table 5: Paired McNemar’s test at seed 42

Policy	MC Acc.	Macro-P	Macro-R	Macro-F1
B1 Static-only	0.9076	0.9765	0.6264	0.7022
B2 Dynamic-only	0.8950	0.8893	0.7502	0.7888
B3 Memory-only	0.5001	0.0833	0.0562	0.0383
B7 Exhaustive	0.9338	0.9806	0.7341	0.8204
H0 HMAS	0.9890	0.9887	0.9890	0.9748
H1 HMAS + LLM review	0.9419	0.9473	0.9361	0.9417

Table 6: Family attribution, default threshold, single run (seed 42). B2/B3 rows are the dynamic/memory domain results from

Class	P	R	F1	Support
Benign	0.9896	0.9994	0.9945	6209
GandCrab	0.9977	0.9873	0.9925	1736
LockBit	0.9905	0.9748	0.9826	1708
Zeppelin	0.9736	0.9946	0.9840	1299
WannaCry	1.0000	0.9947	0.9974	379
Cerber	0.9898	0.9198	0.9535	212

Table 7: Per-class results for H0, all 17 classes, default threshold,

n=12{,}439

(recomputed from the reported multiclass confusion matrix; exactly reproduces the file’s reported macro-averages).

Policy	Mean cost (units)	Cost redn. vs. exhaustive	Mean latency (ms)	p95 latency (ms)
B1 Static-only	4.00	66.67%	15.67	16.08
B6 Static+Dynamic	10.00	16.67%	245.52	255.50
B7 Exhaustive	12.00	0%	249.28	259.29
H0 HMAS	6.65	44.6%	111.34	218.30

Table 8: Internal cost units and mean/p95 inference latency, default threshold, seed 42

Agent pool size	$n$ (cases)	Mean latency (ms)	Throughput (samples/s)
3	7209	37.47	26.69
6	4457	213.47	4.68
7	773	211.34	4.73

Table 9: Throughput observations by agent-pool-size bucket

Short-circuit error rate (ransomware missed at the static stage): 71/6230 = 1.14%
Route	$n$	Share	Acc. on route	FN on route	FP on route
Static only	7209	57.95%	0.9900	71	1
Static+Dynamic	4457	35.83%	0.9493	225	1
Full pipeline (+Memory)	773	6.21%	0.4386	432	2

Table 10: Routing distribution and per-route outcomes for H0

Figure 7: Measured binary accuracy by escalation route, H0, seed 42. Accuracy falls sharply on the 6.21% of cases that reach the full pipeline.

Row label (as logged)	$n$	Acc.	TN	FP	FN/TP
“static” (= B1 static-only)	12439	0.9130	6203	6	1076/5154
“dynamic” (= B2 dynamic-only)	12439	0.9329	6001	208	627/5603
“memory” (= B3 memory-only)	12439	0.4994	6209	0	6227/3
$-$ adaptive routing (= B7 exhaustive)	12439	0.9401	6205	4	741/5489
$-$ LLM review (= H0)	12439	0.9412	6205	4	728/5502

Table 11: Leave-one-component-out

$\lambda$	B1 Static-only	B6 S+D	B7 Exhaustive	H0 HMAS
0.0	0.9130	0.9647	0.9401	0.9412
0.1	0.5130	-0.0353	-0.2599	0.2765
0.5	-1.0870	-4.0353	-5.0599	-2.3823
1.0	-3.0870	-9.0353	-11.0599	-5.7058

Table 12: Objective value

\mathrm{Acc}-\lambda\cdot\mathrm{cost}

at selected

\lambda

(seed 42, default threshold).

Quantity	H0 (deterministic)	H1 (LLM-enabled)
Mean cost (internal units)	6.65	14.07
Mean decision latency	111 ms	6,639 ms
p95 decision latency	218 ms	22,751 ms
Invocation rate	–	31.1%
Mean LLM calls per invoked case	–	1.48

Table 13: Cost of enabling the LLM reviewer, default threshold, single run (seed 42,

n=12{,}439

Family	$n$	Accuracy
Benign	22235	0.9964
Zeppelin	4695	0.6062
GandCrab	6204	0.5608
LockBit	6110	0.5525
Sodinokibi	779	0.4878
Cerber	730	0.3411

Table 14: Standalone per-family accuracy. Families with

n<50

(Cyclops,

n{=}2

) omitted as uninformative.

Malware Detection

Malware detection uses static and behavioral evidence to identify malicious software and protect devices, networks, and software ecosystems.

See all 57 papers

Agentic SABRE: An Uncertainty-Aware Neuro-Symbolic Multi-Agent Framework for Adaptive Ransomware Detection
Jul 5, 2026Henry Kabuye, Biju Issac, Jeyamohan NeeraMalware Classification Neuro-Symbolic AI
School of Computer Science, Northumbria University, Newcastle Upon Tyne, UK
RansomTrack: A Hybrid Behavioral Analysis Framework for Ransomware Detection
Apr 9, 2026Busra Caliskan, Ibrahim Gulatas, H. Hakan Kilinc +1Malware Detection Malware Analysis
Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye · Department of Research and Development, Orion Innovation, Istanbul, Türkiye · Department of Research and Development, Turkish Naval Forces, Istanbul, Türkiye
Hardware-Assisted Multi-Stage Ransomware Detection with Adaptive Machine Learning
Oct 11, 2026Salma Aboelmagd, Md. Rakibul Ahasan, Zhixin Pan +2Malware Detection
Florida State University, Tallahassee, FL, USA · Stony Brook University, Stony Brook, NY, USA

Jul 5, 2026J/K move · Enter open · S save

Agentic SABRE: An Uncertainty-Aware Neuro-Symbolic Multi-Agent Framework for Adaptive Ransomware Detection

Henry Kabuye, Biju Issac, Jeyamohan Neera

School of Computer Science, Northumbria University, Newcastle Upon Tyne, UK

Malware Classification Neuro-Symbolic AI Malware Detection

Open paper AskPDF arXiv

Ransomware has evolved into a complex, adaptive, and fast-moving adversary category in which static signatures and monolithic classifiers fail to generalise under concept drift, evasion, and behavioural polymorphism. In this paper, we present Agentic SABRE (Semantic-Behavioural Arbitration for Ransomware Evaluation), an uncertainty-aware, neuro-symbolic, multi-agent framework for adaptive ransomware detection. SABRE fuses semantic, representation-based evidence with behavioural, time-window forensic telemetry and employs Monte Carlo Dropout inference to quantify epistemic uncertainty for each agent. We introduce a decision-layer orchestrator that performs risk- and uncertainty-aware triage using two interpretable thresholds: a risk score and an uncertainty budget. High-confidence, high-risk samples are automatically contained, while uncertain or borderline cases are escalated to human analysts, establishing a flexible computational contract between autonomous response and analyst oversight. To support auditability and trust, SABRE integrates post-hoc explainability mechanisms, including gradient saliency, permutation importance, and counterfactual analysis, enabling both local and global interpretation of agent decisions. Extensive evaluation on RDset and RanSMAP demonstrates that Agentic SABRE preserves perfect discrimination on saturated semantic datasets, with AUC equal to 1.0, while improving robustness under weak behavioural signals. It achieves up to a 4.9 percent relative reduction in false escalations at equal recall while maintaining calibrated predictive uncertainty. Counterfactual analysis further shows that semantic and behavioural decisions can be reversed with bounded perturbation cost, indicating stable and interpretable decision boundaries.

Abstract

Malware Detection

Explore similar work

Agentic SABRE: An Uncertainty-Aware Neuro-Symbolic Multi-Agent Framework for Adaptive Ransomware Detection

RansomTrack: A Hybrid Behavioral Analysis Framework for Ransomware Detection

Hardware-Assisted Multi-Stage Ransomware Detection with Adaptive Machine Learning