DYNAS: A Dynamic Network Penetration Testing Framework Using Deep Hierarchical Reinforcement Learning

Md. Rakibul Hassan; Majedul Islam; Md. Ashraful Alam; Muhammad Iqbal Hossain; Swakkhar Shatabda

cs.CROct 29, 2026

DYNAS: A Dynamic Network Penetration Testing Framework Using Deep Hierarchical Reinforcement Learning

Authors: Md. Rakibul Hassan, Majedul Islam, Md. Ashraful Alam, Muhammad Iqbal Hossain, Swakkhar Shatabda

Organizations: BRAC University, Bangladesh

Abstract

As network scale and uncertainty complicate automated penetration testing, reinforcement learning (RL) environments must support controlled evaluation under defensive and non-stationary conditions. We present DYNAS (Dynamic Network Attack Simulator), which combines parameterized large-scale scenario generation with stateful intrusion detection responses, explicit scan false-positive and false-negative noise, temporary service unavailability, and duration-based action timeouts. We further introduce Pen-DHRL, a recurrent manager--worker policy that selects persistent subgoals using IDS-conditioned global context. We compare Pen-DHRL with GNN, Invariant, and Attention baselines across four scenario variants and multiple training regimes. Under IDS-disabled training followed by IDS-enabled evaluation, Pen-DHRL obtains approximately 64% and 24% higher reward per step on Bridge and Dynamic, respectively, and approximately 26% and 30% more captured hosts on VarA and VarB than the next-best baseline. With all defensive dynamics active from the beginning of training, Pen-DHRL achieves positive mean episode returns in all four scenarios, whereas the flat baselines produce negative mean returns in all 12 architecture--scenario combinations; staged curriculum training produces positive mean returns across all 16 evaluated combinations.

Figures & tables

Feature	NASim	IAPTF	WCE	DynPen	NASimEmu	PenGym	DYNAS
Simulation-Based Training	✓	✓	Conceptual	✓	✓	✓	✓
Structural Scenario Variation	Limited	–	Conceptual	✓	Limited	Limited	✓
Firewall-Aware Reachability	✓	Limited	Conceptual	–	✓	✓	✓
IDS Representation/Response	–	Limited	–	–	–	–	✓
Curriculum Scheduling	–	–	–	–	–	–	✓
Explicit Scan FP/FN Noise	–	–	–	–	–	–	✓

Table 1: Abstraction-level and environmental-feature comparison of RL-based pentesting frameworks

Figure 1: Overview of Dynamic Scenario Generation and HRL Agent Development. Part 1 (left, detailed in Part 1 below) is the scenario pipeline, from the cached template through the NASimEmu runtime. Part 2 (right, detailed in Part 2 below) is the Pen-DHRL agent, from graph encoding through the value and termination heads. Solid arrows carry data and decisions during an episode; dashed arrows carry gradients during the recurrent PPO update.

Figure 2: Pen-DHRL Architecture Implemented in DYNAS. Left: the forward path for one decision. Node features and positional encodings are embedded and passed through the recurrent manager GNN to obtain per-node features and a graph-level global context; the graph-wise mean of the per-host IDS fields is projected and added to that context. The manager either keeps its active goal or draws a new slot from the goal bank, refines it with the global context, and passes the result to a GAT worker that emits a (host, action) decision. Value and termination heads read the concatenated global context, refined goal, and normalized episode progress. Right: the state carried across steps, and the recurrent PPO update, which snapshots that state, replays the recorded decisions with done-masked resets, and applies the clipped policy, value, and entropy losses.

Scenario	$(N,H,S)$	$A_{\text{heuristic}}$	Captured hosts	Episode return
Corp-Dyn	$(39,83,12)$	$314.5$	$36.13$	$49.41$
VarA	$(41,86,12)$	$327.5$	$36.68$	$48.43$

Table 2: Scenario dimensions, heuristic interaction budget, and observed Pen-DHRL outcomes in the defense-disabled setting

Scenario	LinSvc	WinSvc	AnySvc	SensSvc	LinProc	WinProc	UserExp	RootExp	PrivEsc
Corp	3	2	1	1	0	0	4	1	1
Uni	3	2	1	1	0	0	4	1	1
Corp-Dyn	23	8	1	3	6	7	7	1	6
VarA	23	8	1	3	6	7	8	0	6

Table 3: Service and process distribution across network scenarios

Scenario	Subnets	Hosts	Exploits	SensHosts	Density
Corp	8	21	5	7	0.25
Uni	9	27	5	3	0.27
Corp-Dyn	12	83	8	39	0.19
VarA	12	86	8	41	0.19

Table 4: Structural and security characteristics of network scenarios

Metric	Architecture	Training	Testing
Reward/step	GNN	0.27	0.27
	Invariant	0.34	0.33
	Attention	0.26	0.26
	Pen-DHRL	0.49	0.48
Episode return	GNN	29.90	30.02
	Invariant	27.21	27.06

Table 5: Baseline performance on dynamically generated scenarios without IDS or curriculum

		Reward/step			Episode return
Scenario	Model	No	Curr	$\Delta$	No	Curr	$\Delta$
Bridge	Pen-DHRL	$\phantom{-}0.40$	$1.51$	$+1.11$	$\phantom{-}161$	$213$	$+53$
	GNN	$-0.08$	$1.37$	$+1.45$	$-31$	$189$	$+220$
	Invariant	$-1.30$	$1.22$	$+2.52$	$-511$	$206$	$+717$
	Attention	$-1.23$	$1.24$	$+2.47$	$-490$	$205$	$+695$
Dynamic	Pen-DHRL	$\phantom{-}0.54$	$2.25$	$+1.71$	$\phantom{-}217$	$249$	$+32$

Table 6: IDS-enabled evaluation of reward per step and episode return: no-curriculum vs. curriculum. Episode returns are rounded to integers; each

\Delta

uses unrounded values

Scenario	Model	No	Curr	$\Delta$	No	Curr	$\Delta$
		Episode length			Captured hosts
Bridge	Pen-DHRL	400	141	$-259$	24.95	25.72	$+0.77$
	GNN	400	138	$-262$	0.99	23.68	$+22.69$
	Invariant	392	170	$-222$	2.24	25.56	$+23.32$
	Attention	400	165	$-235$	1.07	23.72	$+22.65$
Dynamic	Pen-DHRL	400	113	$-287$	29.59	30.37	$+0.78$

Table 7: IDS-enabled evaluation of episode length and captured hosts: no-curriculum vs. curriculum. Episode lengths are rounded to integers; each

\Delta

uses unrounded values

Scenario	Architecture	Reward/step	Episode return	Ep Len	Capture
Bridge	Pen-DHRL	$\phantom{-}1.15$	$\phantom{-}161.96$	141.20	23.36
	GNN	$\phantom{-}0.18$	$\phantom{-}66.20$	368.86	26.17
	Invariant	$\phantom{-}0.01$	$\phantom{-}5.20$	367.77	25.69
	Attention	$\phantom{-}0.70$	$\phantom{-}120.04$	170.39	17.58
Dynamic	Pen-DHRL	$\phantom{-}1.49$	$\phantom{-}200.17$	134.56	27.80
	GNN	$\phantom{-}0.26$	$\phantom{-}91.32$	356.31	30.59

Table 8: Per-scenario model performance with IDS disabled during training but enabled during evaluation

Automated Penetration Testing

Automated penetration testing uses software agents to find, verify, and assess exploitable weaknesses in systems.

See all 49 papers

NASimJax: A GPU-Accelerated Policy Learning Framework for Penetration Testing
Mar 20, 2026Raphael Simon, José Carrasquel, Elli Makdis Antoun +2RL Benchmarks Partially Observable RL
CISS Department, Royal Military Academy, Belgium · AI Lab, Vrije Universiteit Brussel, Belgium
Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation
Sep 24, 2025Raphael Simon, Pieter Libin, Wim MeesReinforcement Learning Partially Observable RL
Cyber Defence Lab, CISS Department Royal Military Academy · AI Lab, Department of Computer Science Vrije Universiteit Brussel
Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)
Aug 5, 2026Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi +6Autonomous Cyber Defense LLM Red Teaming
University of California, Irvine · Northeastern University · Johns Hopkins University

Mar 20, 2026J/K move · Enter open · S save

NASimJax: A GPU-Accelerated Policy Learning Framework for Penetration Testing

Raphael Simon, José Carrasquel, Elli Makdis Antoun, Wim Mees+1

CISS Department, Royal Military Academy, Belgium · AI Lab, Vrije Universiteit Brussel, Belgium

RL Benchmarks Partially Observable RL Automated Penetration Testing

Open paper AskPDF arXiv

Penetration testing - the practice of simulating cyberattacks to identify vulnerabilities - is a complex sequential decision-making task that is inherently partially observable and features large action spaces. Existing RL simulators for this domain are CPU-bound and fixed to narrow scenarios, making it infeasible to train policies that generalize across networks. We present NASimJax, a JAX-native framework that formulates penetration testing as a Contextual POMDP and introduces a network generation pipeline producing structurally diverse, guaranteed-solvable scenarios. The framework reaches up to 80

\times

higher environment throughput than previous simulators, enabling experiments on larger networks and tractable hyperparameter searches. We provide PPO and PQN baselines and conduct the first systematic evaluation of unsupervised environment design for penetration testing. We find that Prioritized Level Replay and ACCEL handle dense training distributions better than Domain Randomization, and that training on sparser topologies yields an implicit curriculum that improves generalization - even to topologies denser than those seen during training. A recurrent PPO variant confirms that these distributional findings are not an artifact of feed-forward policies. The code is available at: https://github.com/raphsimon/NASimJax.

Abstract

Automated Penetration Testing

Explore similar work

NASimJax: A GPU-Accelerated Policy Learning Framework for Penetration Testing

Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)