FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks
Organizations: Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA
Abstract
Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2 better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.
Figures & tables
| Vulnerability | Severity (CVSS) | Hosts |
|---|---|---|
| WordPress Backup Migration | 9.8 | PublicWeb1, PublicWeb2 |
| Weak SSH credentials | N/A | NTP, DB, WEB |
| Weak sudo version | N/A | NTP, DB, WEB |
| World-writable cron-as-root | N/A | PublicWeb1, PublicWeb2 |
| Objective | Targets | Parameter blocks |
|---|---|---|
| World model | , ; latent priors | |
| Opponent | Next attacker action and target | Encoder , heads |
| Actor-critic | Imagined returns | Actor , critic |
| Method | Avoidant | Enticed |
|---|---|---|
| FOE-Dreamer | ||
| Rainbow | ||
| IQN | ||
| Expert | ||
| Random | ||
| Sleep |
| Configuration | Avoidant | Enticed |
|---|---|---|
| Full (FOE-Dreamer) | ||
| Full Factor | ||
| Full Factor Opponent |
| Input latent | (U-rand) | (U-burst) |
|---|---|---|
| 0.72 | 0.79 | |
| 0.14 | 0.18 |
| Input latent | Accuracy |
|---|---|
| 0.21 | |
| 0.71 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Description |
|---|---|
| Observation at time (per-host telemetry vector) | |
| Defender action at time | |
| Scalar reward at time | |
| Controllable stochastic latent (action-conditioned) | |
| Exogenous stochastic latent (action-independent) | |
| Opponent embedding at time |
| Hyperparameter | Range (lower, upper) | Selected |
|---|---|---|
| Controllable latent dim ( ) | 32, 256 | 128 |
| Exogenous latent dim ( ) | 32, 256 | 128 |
| Opponent embedding dim ( ) | 16, 128 | 64 |
| Opponent sliding window ( ) | 5, 20 | 10 |
| Imagination horizon ( ) | 10, 50 | 30 |
| KL coefficient | 0.1, 2.0 | 1.0 |
Autonomous Cyber Defense
Autonomous cyber defense uses AI agents to detect, investigate, and contain attacks while balancing operational safety, latency, and human oversight.
See all 76 papers