Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents

Sabrina Saika; Yinuo Du; Aritran Piplai

cs.CRSep 30, 2026

Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agents

Authors: Sabrina Saika, Yinuo Du, Aritran Piplai

Organizations: The University of Texas at El Paso, El Paso, TX, USA

Abstract

Cyber attack agents are typically trained and evaluated within a single simulator, making it unclear whether learned policies transfer beyond the environments in which they were developed. This limitation hinders both deployment and fair comparison, as cyber simulators differ substantially in their state representations, observation models, and action spaces. In this paper, we study policy transfer across cyber environments and argue that simulator-to-simulator and simulator-to-real transfer can be viewed as instances of the same underlying alignment problem. We propose a framework that separates state alignment from action translation, enabling a policy trained in one environment to operate in another without retraining. We evaluate transfer across four cyber platforms, CyberBattleSim, NetSecGame, CyberWheel, and NASim, including emulated deployments in NASim. Our experiments show that zero-shot transfer is feasible, fully preserving source-policy performance in closely aligned environments and achieving 45.2% win rates when transferring policies whose source performance is 60.5%. In emulated virtual machine environments, transferred policies exhibit a Jensen-Shannon divergence of 0.085 from native policies, indicating strong behavioral similarity. Code and benchmarks are available at: https://anonymous.4open.science/r/RL-Transfer-between-env-4F47/.

Figures & tables

Figure 1: Main architecture of the proposed framework. A policy trained in a source cyber environment is transferred to a target environment through state alignment and action translation, enabling zero-shot execution across simulators and emulated systems.

		Observation		Action
Env.	Train-to-converge	Dim	Decision-relevance	Space	Type	Task horizon	Configurable
CW	PPO steps (fast)	78/512	High (compact kill-chain)	13 discrete	Informative + exploitative	7 kill-chain phases/host	Yes (YAML hosts/subnets)
NSG	Fast ( $\approx$ CW)	78	High (all features relevant)	12 slot actions	Informative + exploitative	Discovery $\to$ impact, $\leq$ 25 steps	Fixed topology
CBS	Slow	512	Low (many irrelevant features)	9 discrete	Exploitative (discovery supplied)	Short: ${\sim}3$ actions, ${\sim}8$ steps/8 nodes	Yes, harder than CW
Emulation (NASim)	n/a	—	n/a	Concrete tool invocations	Informative + exploitative	n/a	VMs, complex setup

Table 1: Environments and their characteristics

Ph.	Label	CBS $\psi$	NSG $\psi$	NASim $\psi$	Notes
0	Network disc.	local_exploit (frontier node)	ScanNetwork()	ServiceScan (subnet)	Slot has no effect; $\psi$ is constant
1	Service disc.	local_exploit (frontier node)	FindServices( $h$ )	OSScan / ProcScan	Slot has no effect; $\psi$ is constant
2	Exploitation	Connect to target node	ExploitService( $h$ )	SubnetScan	Held until host is controlled
3	Post-exploit	local_exploit (owned node)	FindData( $h$ )	Exploit( $h$ )
4	Exfiltration	Cycles all vulnerabilities	ExfiltrateData( $h$ , C&C)	PrivEsc( $h$ ) Exfiltrate Data	Requires data present
5+	Terminal	—	No-op	No-op	NSG/NASim only

Table 2: Kill-chain phase mapping used by

\psi

for the CW

\rightarrow

CBS, CW

\rightarrow

NSG, and CW

\rightarrow

NASim transfer pairs.

Figure 2: t-SNE projection of CW (blue) and NSG (orange) observations before (left) and after (right) the DAPN encoder. After encoding, the two domain clouds overlap substantially, confirming that the adversarial training closes the distributional gap between simulators.

Condition	CW Win%	NSG Win%	Win Steps	Mean Return
Random Policy	—	0.2%	—	$-24.8$
Zero-Shot Transfer	21.7%	13.5%	7.3	$-9.0$
Zero-Shot Transfer + Feature Eng.	99.1%	47.4%	7.3	$30.9$
Zero-Shot Transfer + Feature Eng. + DAPN	60.5%	45.2%	7.2	$28.2$

Table 3: NSG evaluation results. Max 25 steps, 3 seeds

\times

500 episodes.

Condition	Win Rate	Nodes Owned	Steps to 1st win	Mean Return
Random Policy	$0\%$	$2.60$	$21.9$	$336$
Zero Shot + Feature Eng.	$0\%$	$0.00$	—	$19$
Zero Shot + Feature Eng. +DAPN	$\mathbf{100\%}$	$\mathbf{8.00}$	$\mathbf{13.1}$	$\mathbf{6{,}024}$

Table 4: Three conditions evaluated on CBS chain-12, win

=8

nodes owned,

20

episodes.

Figure 3: Action-type distribution over winning trajectories. Each panel compares a transferred policy against the NASim emulator policy across the five NASim action types with Jensen–Shannon divergence reported per panel. Translated CW $\rightarrow$ NASim (JS $=0.087$ ). Original CW vs. NASim emulator policy, no win (JS $=0.111$ ). Translated NSG $\rightarrow$ NASim (JS $=0.085$ ).

Explore similar work

Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment
May 26, 2026Gengyue Han, Yiheng FengSim-To-Real Reinforcement Learning Sim-To-Real Gap
Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, USA · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA
Provably Safe Sim-to-Real Transfer
Sep 1, 2026Tingting Ni, Maryam KamgarpourSim-To-Real Reinforcement Learning Simulation-Based Reinforcement Learning
SYCAMORE, EPFL
Abstract Sim2Real through Approximate Information States
Apr 16, 2026Yunfu Deng, Yuhao Li, Josiah P. HannaSim-To-Real Reinforcement Learning Sim-To-Real Gap
Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003 USA

May 26, 2026, cs.LGJ/K move · Enter open · S save

Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

Gengyue Han, Yiheng Feng

Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette, USA · Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA

Sim-To-Real Reinforcement Learning Sim-To-Real Gap Offline Reinforcement Learning Markov Decision Processes

Open paper AskPDF arXiv

Due to limited resources and public safety concerns, deep reinforcement learning (RL) agents for many cyber-physical systems (e.g., autonomous vehicles) are first trained in simulators. However, when deployed in real world environments, they often suffer from performance degradation or safety violations because of the inevitable Sim2Real gap. Existing zero-shot approaches, such as robust safe RL and domain randomization, mitigate this issue but typically at the cost of degraded performance or residual safety risks when experiencing unmodeled system dynamics. To address these limitations, we propose a novel reinforcement learning framework that enables safe and efficient policy transfer via probabilistic latent embeddings and dynamic policy adaptation. We consider a family of Constrained Markov Decision Processes (CMDPs) under different environment contexts. By leveraging latent context variable in meta-RL, the proposed framework infers the latent representation of the environment from simulated experiences. Furthermore, it incorporates a distributional RL formulation, which allows risk levels of the deployed policy to be adjusted dynamically, based on the estimation accuracy of the latent context variable. This strategy promotes safety at the early deployment stage and improves efficiency through fast policy adaptation under the Sim2Real gap.

Abstract

Explore similar work

Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

Provably Safe Sim-to-Real Transfer

Abstract Sim2Real through Approximate Information States