SubstratumGraphEnv: Reinforcement Learning Environment (RLE) for Modeling System Attack Paths

Bahirah Adewunmi; Edward Raff; Sanjay Purushotham

cs.CRMar 2, 2026

SubstratumGraphEnv: Reinforcement Learning Environment (RLE) for Modeling System Attack Paths

Authors: Bahirah Adewunmi, Edward Raff, Sanjay Purushotham

Organizations: University of Maryland, Baltimore County · CrowdStrike

Abstract

Automating network security analysis, particularly the identification of potential attack paths, presents significant challenges. Due in part to the sequential, interconnected, and evolutionary nature of system events which most artificial intelligence (AI) techniques struggle to model effectively. This paper proposes a Reinforcement Learning (RL) environment generation framework that simulates the sequence of processes executed on a Windows operating system, enabling dynamic modeling of malicious processes on a system. This methodology models operating system state and transitions using a graph representation. This graph is derived from open-source System Monitor (Sysmon) logs. To address the variety in system event types, fields, and log formats, a mechanism was developed to capture and model parent-child processes from Sysmon logs. A Gymnasium environment (SubstratumGraphEnv) was constructed to establish the perceptible basis for an RL environment, and a customized PyTorch interface was also built (SubstratumBridge) to translate Gymnasium graphs into Deep Reinforcement Learning (DRL) observations and discrete actions. Graph Convolutional Networks (GCNs) concretize the graph's local and global state, which feed the distinct policy and critic heads of an Advantage Actor-Critic (A2C) model. This work's central contribution lies in the design of a novel deep graphical RL environment that automates translation of sequential user and system events, furnishing crucial context for cybersecurity analysis. This work provides a foundation for future research into shaping training parameters and advanced reward shaping, while also offering insight into which system events attributes are critical to training autonomous RL agents.

Figures & tables

Figure 1: This partial network snapshot of the BRAWL dataset reveals the sparse and disassortative nature of the system’s process graph. The node labels—ranging from standard Windows processes like svchost to strings from the ”Fresh Prince” theme—are raw identifiers extracted directly from the dataset’s Sysmon logs. While frequent processes like powershell have numerous connections (thick edges), the low clustering demonstrates that an RL agent cannot rely on dense local patterns, but must instead learn to navigate these non-intuitive, sparse pathways to identify the full attack chain.

Figure 2: This snapshot of Cerberus Traces reveals a dense, complex process graph with a few processes, like pythonw , acting as central hubs for activity. Despite this high connectivity, the network’s low clustering shows that critical attack sequences are not confined to local, dense event patterns. Therefore, the RL environment must guide the agent through a large, distributed state space to discover significant sequential connections.

Key Degree (k)	Average Nearest Neighbor Degree ( ${k}_{nn}$ )	Key Degree (k)	Average Nearest Neighbor Degree ( ${k}_{nn}$ )
Cerberus Traces		BRAWL
1	487.94	1	0.10
2	103.11	2	0.77
3	152.66	3	0.46
4	55.98	4	0.67
15	63.08	5	0.34

Table 1: The highly varied, often low Average Nearest Neighbor Degree and the disassortative patterns where high-degree nodes connect to low-degree neighbors, demonstrate that rewards cannot simply rely on local, dense clusters of events

Process/Node Name	In-Degree Centrality Value	Dataset
conhost	0.13600	BRAWL
Cmd	0.06800	BRAWL
Powershell	0.05800	BRAWL
Reg	0.05800	BRAWL
Net	0.04900	BRAWL
WMIC	0.04900	BRAWL

Table 2: The significantly higher centrality scores in BRAWL (e.g., conhost , cmd ) versus Cerberus Traces reveal two distinct interaction patterns: centralized in BRAWL and distributed in Cerberus. This structural difference confirms the need for a reward function that can target dispersed critical nodes, not just single, highly connected targets.

Panel of Figure 3 — Figure 3: High-level pipeline from raw Sysmon relations to TorchRL-ready graph observations.

Figure 4: Value loss versus training steps on the Cerberus DRL environment. Decreasing value loss indicates that critic is learning to predict value accurately

	BRAWL	Cerberus
Avg Reward (mean)	-18.42	-4.66
Avg Segment (std)	63.83	0.76
Avg Policy Loss (mean)	0.09	0
Avg Policy Loss (std)	26.42	0
Avg Value Loss (mean)	1.21E+11	4.58E+08
Avg Value Loss (std)	1.01E+13	6.87E+08

Table 3: Averaged Parallelized Worker Training Metrics for the DRL Environments. The Cerberus environment demonstrates feasibility and stability with negligible loss variance. In contrast, the high loss standard deviations in BRAWL (

1.01\times 10^{13}

) confirm that its sparse graph structure presents a challenging stability problem for the DRL agent.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions
Apr 10, 2026Igor JankowskiAutonomous Cyber Defense Cybersecurity
CSLE: A Reinforcement Learning Platform for Autonomous Security Management
Apr 16, 2026Kim HammarReinforcement Learning Autonomous Cyber Defense
Division of Network and Systems Engineering, KTH Royal Institute of Technology, Sweden
Building Better Environments for Autonomous Cyber Defence
Apr 9, 2026Chris Hicks, Elizabeth Bates, Shae McFadden +12Autonomous Cyber Defense Cybersecurity
The Alan Turing Institute · University College London · King’s College London +4

Apr 10, 2026J/K move · Enter open · S save

NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions

Igor Jankowski

Autonomous Cyber Defense Cybersecurity AI Agent Benchmarks Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Training reinforcement-learning agents for cyber defense requires an environment that reflects the operational setting: noisy, partial observations, several defenders coordinating across a network, and an adaptive adversary realized through self-play. We present NetForge RL, a multi-agent environment for this setting on procedurally generated enterprise and operational-technology (OT) networks. A red agent compromises hosts with partial observability; three zone-split blue agents defend from synthetic SIEM telemetry, Windows/Sysmon event logs encoded into dense embeddings rather than a ground-truth state vector. The environment follows the PettingZoo parallel API with fixed-shape observations and MITRE ATT&CK-mapped actions, ships five scenarios with named difficulty presets and a held-out evaluation split, and replays deterministically under a seed. A JAX backend vectorizes a reduced transition core, reaching 2.5 x 10^5 environment-steps/s at batch 4096 on CPU, as a fast surrogate for training-loop iteration. Alongside the environment we provide reference baselines (scripted, a JAX IPPO trainer, and a self-play tournament), six diagnostic probes that each measure one defensive skill, and an evaluation runner reporting 95% confidence intervals. We describe the reproducibility engineering behind the environment and include a responsible-use statement.

Abstract

Reinforcement Learning

Explore similar work

NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions

CSLE: A Reinforcement Learning Platform for Autonomous Security Management

Building Better Environments for Autonomous Cyber Defence