Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2× better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.
Figures & tables
Fig. 1: Agent-environment interface. The C2 server provides bidirectional translation: abstract defensive tactics become concrete network operations (downward), and distributed sensor data becomes structured observations (upward). Communication occurs over gRPC with Protocol Buffers.
Vulnerability
Severity (CVSS)
Hosts
WordPress Backup Migration
9.8
PublicWeb1, PublicWeb2
Weak SSH credentials
N/A
NTP, DB, WEB
Weak sudo version
N/A
NTP, DB, WEB
World-writable cron-as-root
N/A
PublicWeb1, PublicWeb2
TABLE I: CVE-class vulnerabilities deployed in the testbed.
Fig. 2: FOE-Dreamer architecture overview.
Objective
Targets
Parameter blocks
World model
ot , rt ; latent priors
(θ,ϕ,ψ,η,ζ)
Opponent
Next attacker action and target
Encoder ξ , heads ω
Actor-critic
Imagined returns
Actor π , critic v
TABLE II: Objectives and directly supervised parameter blocks.
Method
Avoidant
Enticed
FOE-Dreamer
−48±7
−38±5
Rainbow
−102±20
−76±17
IQN
−118±26
−84±21
Expert
−138±18
−88±15
Random
−152±28
−242±38
Sleep
−178±24
−285±32
TABLE III: Final episode loss (mean ± std over 5 seeds) after 3 days of live training. Lower is worse. Sleep, Random, and Expert-Inspired Heuristic are reference baselines; Rainbow and IQN are learned baselines.
Fig. 3: Episode loss vs. training day across baselines and FOE-Dreamer, under both attacker profiles.
Configuration
Avoidant
Enticed
Full (FOE-Dreamer)
−48±7
−38±5
Full − Factor
−62±12
−48±11
Full − Factor − Opponent
−78±16
−58±14
TABLE IV: Ablation matrix. Final episode loss (mean ± std over 5 seeds) after 3 days. Lower is worse. Full − Factor − Opponent is the single-latent Dreamer baseline.
Input latent
R2 (U-rand)
R2 (U-burst)
ut
0.72
0.79
zt
0.14
0.18
TABLE V: Linear-decoding R2 for user activity from each latent. Higher means stronger linear relationship.
Input latent
Accuracy
ut
0.21
zt
0.71
TABLE VI: Defender-action prediction accuracy from each latent.
Fig. 4: t-SNE projection of the opponent embedding τ at a fixed step in each episode, colored by attacker profile. The two profiles occupy clearly separated regions of the embedding space.
Fig. 5: Action-prediction accuracy of the opponent encoder over training, by attacker profile. Accuracy rises monotonically as the encoder learns to summarize defender-observable signals.
Policy-based reinforcement learning (RL) approaches have produced promising results for autonomous cyber defense; however, they are sample-inefficient in settings where defenders must respond under delayed, partial observations with actions from large action spaces. While large language models (LLMs) may reason semantically about security state space, high latency and trust assumptions prevent attractive in-line deployment models. We introduce Ask the Expert, a training-time guidance framework which first summarizes hard cyber-defense states, then intermittently queries an LLM for host-level defensive recommendations via a constrained action interface, and finally transforms those recommendations into tiered reward shaping for use with PPO. Because the LLM is discarded after training, deployment is a pure RL policy. Across TTCP CAGE CC1 and CC2 and both attacker types, this asymmetric design improves sample efficiency over PPO and outperforms the evaluated potential-based reward shaping (PBRS) baselines, while retaining the strongest terminal mean and requiring no LLM dependency at deployment time.
Fernando Martinez, Abhishek Satyam, Tao Li +3
Department of Computer and Information Sciences, Fordham University, New York, USA · Department of Systems Engineering, Stevens Institute of Technology, New Jersey, USA
Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the "world" in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and multimodal representations of the defended network. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k-15.8k environment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under topology-dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability.
Training reinforcement-learning agents for cyber defense requires an environment that reflects the operational setting: noisy, partial observations, several defenders coordinating across a network, and an adaptive adversary realized through self-play. We present NetForge RL, a multi-agent environment for this setting on procedurally generated enterprise and operational-technology (OT) networks. A red agent compromises hosts with partial observability; three zone-split blue agents defend from synthetic SIEM telemetry, Windows/Sysmon event logs encoded into dense embeddings rather than a ground-truth state vector. The environment follows the PettingZoo parallel API with fixed-shape observations and MITRE ATT&CK-mapped actions, ships five scenarios with named difficulty presets and a held-out evaluation split, and replays deterministically under a seed. A JAX backend vectorizes a reduced transition core, reaching 2.5 x 10^5 environment-steps/s at batch 4096 on CPU, as a fast surrogate for training-loop iteration. Alongside the environment we provide reference baselines (scripted, a JAX IPPO trainer, and a self-play tournament), six diagnostic probes that each measure one defensive skill, and an evaluation runner reporting 95% confidence intervals. We describe the reproducibility engineering behind the environment and include a responsible-use statement.