DeepStage: Learning Autonomous Defense Policies Against Multi-Stage APT Campaigns

Trung V. Phan; Tri Gia Nguyen; Thomas Bauschert

cs.CRMar 17, 2026

DeepStage: Learning Autonomous Defense Policies Against Multi-Stage APT Campaigns

Authors: Trung V. Phan, Tri Gia Nguyen, Thomas Bauschert

Organizations: Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany · Department of Information Assurance, FPT University, Da Nang 50509, Vietnam

Abstract

This paper presents DeepStage, a deep reinforcement learning (DRL) framework for adaptive and stage-aware defense against Advanced Persistent Threats (APTs). The enterprise environment is formulated as a partially observable Markov decision process (POMDP), in which host provenance and network telemetry are fused into unified provenance graphs. Building on our prior work (StageFinder), DeepStage employs a graph neural network encoder and an LSTM-based stage estimator to infer probabilistic attacker stages aligned with the MITRE ATT&CK framework. The resulting stage beliefs, together with graph embeddings, are used to guide a hierarchical Proximal Policy Optimization (PPO) agent that selects defense actions across monitoring, access control, containment, and remediation. Experiments in a realistic enterprise testbed with CALDERA-driven APT playbooks show that DeepStage achieves an average F1-score of 0.887 and a mitigation success rate of 84.7%, outperforming a risk-aware DRL baseline by 21.8% in F1-score and 16.2% in mitigation success. The results demonstrate effective stage-aware and cost-efficient autonomous cyber defense.

Figures & tables

Figure 1: Data and control flow of the proposed DeepStage framework.

Meta-policy	Sub-policy	Deployment in Testbed	Cost $C(a)$
$\mathcal{A}_{\text{mon}}$	$a_{0}$ : Maintain baseline monitoring	Keep auditd.conf , CamFlow, and Zeek agent in baseline mode	0.01
	$a_{1}$ : Increase host logging level	Modify auditd.conf , CamFlow, and Zeek agent policies to capture process and socket events	0.05
	$a_{2}$ : Activate Deep Packet Inspection	Temporarily enable Zeek full packet capture on target subnets	0.10
	$a_{3}$ : Memory/process snapshot	Use volatility or psrecord via management API to collect runtime memory dump	0.20
	$a_{4}$ : Deploy honeypot redirect	Deploy Cowrie SSH honeypot or redirect via iptables DNAT	0.25
	$a_{5}$ : Log enrichment with threat intel	Query local MISP feed and annotate alerts with TTP tags in provenance database	0.10

Table I: Set of practical sub-policies deployable. Cost

C(a)

denotes the normalized operational impact.

Parameter	Value / Range	Description
Actor network	3-layer MLP	Input: fused graph embedding ( $g_{t}$ ) + stage belief ( $p_{t}$ ); Output: probability over meta/sub-policy actions.
Critic network	3-layer MLP	Estimates value $V(o_{t})$ for baseline; shared first layer with actor.
Optimizer	Adam ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )	Adaptive optimization for actor and critic updates.
Learning rate ( $\eta$ )	$3\times 10^{-4}$ (decay 0.99 per epoch)	Stable step size for PPO gradient updates.
Discount factor ( $\gamma$ )	$0.99$	Long-term credit assignment for delayed APT mitigation rewards.
GAE parameter ( $\lambda_{\text{GAE}}$ )	$0.95$	Balances bias and variance in advantage estimation.

Table II: PPO training parameters for the DeepStage DRL-based APT defense.

Method	Recon	InitAcc	PrivEsc	LatMov	C2	Exfil	Avg-F1	Mitigation Success (%)
Risk-Aware DRL	0.75	0.73	0.70	0.71	0.76	0.72	0.728	68.5
DeepStage-Unaware	0.82	0.79	0.76	0.78	0.83	0.80	0.797	76.7
DeepStage	0.91	0.88	0.85	0.87	0.92	0.89	0.887	84.7

Table III: Per-stage F1-score and overall defense effectiveness comparison.

Figure 2: Cost–effectiveness frontiers illustrating normalized security gain versus cumulative action cost.

Figure 3: Training convergence of hierarchical PPO across methods.

Figure 4: Defense responsiveness over APT stage transitions.

Autonomous Cyber Defense

Autonomous cyber defense uses AI agents to detect, investigate, and contain attacks while balancing operational safety, latency, and human oversight.

See all 69 papers

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns
Mar 22, 2026Trung V. Phan, Thomas BauschertAutonomous Cyber Defense Reinforcement Learning
Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany
Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning
Apr 6, 2026Yiyao Zhang, Diksha Goel, Hussain AhmadAdversarial MARL Autonomous Cyber Defense
Adelaide University, Australia · CSIRO’s Data61, Australia
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber Defense LLM Red Teaming
University of California, Irvine · Northeastern University · Johns Hopkins University

Mar 22, 2026J/K move · Enter open · S save

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Trung V. Phan, Thomas Bauschert

Chair of Communication Networks, Technische Universität Chemnitz, 09126 Chemnitz, Germany

Autonomous Cyber Defense Reinforcement Learning Interpretability in RL Explainable Artificial Intelligence

Open paper AskPDF arXiv

Advanced Persistent Threats (APTs) are stealthy, multi-stage attacks that require adaptive and timely defense. While deep reinforcement learning (DRL) enables autonomous cyber defense, its decisions are often opaque and difficult to trust in operational environments. This paper presents DeepXplain, an explainable DRL framework for stage-aware APT defense. Building on our prior DeepStage model, DeepXplain integrates provenance-based graph learning, temporal stage estimation, and a unified XAI pipeline that provides structural, temporal, and policy-level explanations. Unlike post-hoc methods, explanation signals are incorporated directly into policy optimization through evidence alignment and confidence-aware reward shaping. To the best of our knowledge, DeepXplain is the first framework to integrate explanation signals into reinforcement learning for APT defense. Experiments in a realistic enterprise testbed show improvements in stage-weighted F1-score (0.887 to 0.915) and success rate (84.7% to 89.6%), along with higher explanation confidence (0.86), improved fidelity (0.79), and more compact explanations (0.31). These results demonstrate enhanced effectiveness and trustworthiness of autonomous cyber defense.

Abstract

Autonomous Cyber Defense

Explore similar work

DeepXplain: XAI-Guided Autonomous Defense Against Multi-Stage APT Campaigns

Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)