Automating network security analysis, particularly the identification of potential attack paths, presents significant challenges. Due in part to the sequential, interconnected, and evolutionary nature of system events which most artificial intelligence (AI) techniques struggle to model effectively. This paper proposes a Reinforcement Learning (RL) environment generation framework that simulates the sequence of processes executed on a Windows operating system, enabling dynamic modeling of malicious processes on a system. This methodology models operating system state and transitions using a graph representation. This graph is derived from open-source System Monitor (Sysmon) logs. To address the variety in system event types, fields, and log formats, a mechanism was developed to capture and model parent-child processes from Sysmon logs. A Gymnasium environment (SubstratumGraphEnv) was constructed to establish the perceptible basis for an RL environment, and a customized PyTorch interface was also built (SubstratumBridge) to translate Gymnasium graphs into Deep Reinforcement Learning (DRL) observations and discrete actions. Graph Convolutional Networks (GCNs) concretize the graph's local and global state, which feed the distinct policy and critic heads of an Advantage Actor-Critic (A2C) model. This work's central contribution lies in the design of a novel deep graphical RL environment that automates translation of sequential user and system events, furnishing crucial context for cybersecurity analysis. This work provides a foundation for future research into shaping training parameters and advanced reward shaping, while also offering insight into which system events attributes are critical to training autonomous RL agents.
Figures & tables
Figure 1: This partial network snapshot of the BRAWL dataset reveals the sparse and disassortative nature of the system’s process graph. The node labels—ranging from standard Windows processes like svchost to strings from the ”Fresh Prince” theme—are raw identifiers extracted directly from the dataset’s Sysmon logs. While frequent processes like powershell have numerous connections (thick edges), the low clustering demonstrates that an RL agent cannot rely on dense local patterns, but must instead learn to navigate these non-intuitive, sparse pathways to identify the full attack chain.
Figure 2: This snapshot of Cerberus Traces reveals a dense, complex process graph with a few processes, like pythonw , acting as central hubs for activity. Despite this high connectivity, the network’s low clustering shows that critical attack sequences are not confined to local, dense event patterns. Therefore, the RL environment must guide the agent through a large, distributed state space to discover significant sequential connections.
Cerberus Traces
BRAWL
Key Degree (k)
Average Nearest Neighbor Degree ( knn )
Key Degree (k)
Average Nearest Neighbor Degree ( knn )
1
487.94
1
0.10
2
103.11
2
0.77
3
152.66
3
0.46
4
55.98
4
0.67
15
63.08
5
0.34
Table 1: The highly varied, often low Average Nearest Neighbor Degree and the disassortative patterns where high-degree nodes connect to low-degree neighbors, demonstrate that rewards cannot simply rely on local, dense clusters of events
Process/Node Name
In-Degree Centrality Value
Dataset
conhost
0.13600
BRAWL
Cmd
0.06800
BRAWL
Powershell
0.05800
BRAWL
Reg
0.05800
BRAWL
Net
0.04900
BRAWL
WMIC
0.04900
BRAWL
Table 2: The significantly higher centrality scores in BRAWL (e.g., conhost , cmd ) versus Cerberus Traces reveal two distinct interaction patterns: centralized in BRAWL and distributed in Cerberus. This structural difference confirms the need for a reward function that can target dispersed critical nodes, not just single, highly connected targets.
Figure 3: High-level pipeline from raw Sysmon relations to TorchRL-ready graph observations.
Figure 4: Value loss versus training steps on the Cerberus DRL environment. Decreasing value loss indicates that critic is learning to predict value accurately
BRAWL
Cerberus
Avg Reward (mean)
-18.42
-4.66
Avg Segment (std)
63.83
0.76
Avg Policy Loss (mean)
0.09
0
Avg Policy Loss (std)
26.42
0
Avg Value Loss (mean)
1.21E+11
4.58E+08
Avg Value Loss (std)
1.01E+13
6.87E+08
Table 3: Averaged Parallelized Worker Training Metrics for the DRL Environments. The Cerberus environment demonstrates feasibility and stability with negligible loss variance. In contrast, the high loss standard deviations in BRAWL ( 1.01×1013 ) confirm that its sparse graph structure presents a challenging stability problem for the DRL agent.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Training reinforcement-learning agents for cyber defense requires an environment that reflects the operational setting: noisy, partial observations, several defenders coordinating across a network, and an adaptive adversary realized through self-play. We present NetForge RL, a multi-agent environment for this setting on procedurally generated enterprise and operational-technology (OT) networks. A red agent compromises hosts with partial observability; three zone-split blue agents defend from synthetic SIEM telemetry, Windows/Sysmon event logs encoded into dense embeddings rather than a ground-truth state vector. The environment follows the PettingZoo parallel API with fixed-shape observations and MITRE ATT&CK-mapped actions, ships five scenarios with named difficulty presets and a held-out evaluation split, and replays deterministically under a seed. A JAX backend vectorizes a reduced transition core, reaching 2.5 x 10^5 environment-steps/s at batch 4096 on CPU, as a fast surrogate for training-loop iteration. Alongside the environment we provide reference baselines (scripted, a JAX IPPO trainer, and a self-play tournament), six diagnostic probes that each measure one defensive skill, and an evaluation runner reporting 95% confidence intervals. We describe the reproducibility engineering behind the environment and include a responsible-use statement.
Reinforcement learning is a promising approach to autonomous and adaptive security management in networked systems. However, current reinforcement learning solutions for security management are mostly limited to simulation environments and it is unclear how they generalize to operational systems. In this paper, we address this limitation by presenting CSLE: a reinforcement learning platform for autonomous security management that enables experimentation under realistic conditions. Conceptually, CSLE encompasses two systems. First, it includes an emulation system that replicates key components of the target system in a virtualized environment. We use this system to gather measurements and logs, based on which we identify a system model, such as a Markov decision process. Second, it includes a simulation system where security strategies are efficiently learned through simulations of the system model. The learned strategies are then evaluated and refined in the emulation system to close the gap between theoretical and operational performance. We demonstrate CSLE through four use cases: flow control, replication control, segmentation control, and recovery control. Through these use cases, we show that CSLE enables near-optimal security management in an environment that approximates an operational system.
Kim Hammar
Division of Network and Systems Engineering, KTH Royal Institute of Technology, Sweden
In November 2025, the authors ran a workshop on the topic of what makes a good reinforcement learning (RL) environment for autonomous cyber defence (ACD). This paper details the knowledge shared by participants both during the workshop and shortly afterwards by contributing herein. The workshop participants come from academia, industry, and government, and have extensive hands-on experience designing and working with RL and cyber environments. While there is now a sizeable body of literature describing work in RL for ACD, there is nevertheless a great deal of tradecraft, domain knowledge, and common hazards which are not detailed comprehensively in a single resource. With a specific focus on building better environments to train and evaluate autonomous RL agents in network defence scenarios, including government and critical infrastructure networks, the contributions of this work are twofold: (1) a framework for decomposing the interface between RL cyber environments and real systems, and (2) guidelines on current best practice for RL-based ACD environment development and agent evaluation, based on the key findings from our workshop.
Chris Hicks, Elizabeth Bates, Shae McFadden +12
The Alan Turing Institute · University College London · King’s College London +4