Evaluating Generalization Mechanisms in Autonomous Cyber Attack Agents

Ondřej Lukáš; Jihoon Shin; Emilia Rivas; Diego Forni; Maria Rigaki; Carlos Catania; Aritran Piplai; Christopher Kiekintveld

cs.CRMar 6, 2026

Evaluating Generalization Mechanisms in Autonomous Cyber Attack Agents

Authors: Ondřej Lukáš, Jihoon Shin, Emilia Rivas, Diego Forni, Maria Rigaki, Carlos Catania, Aritran Piplai, Christopher Kiekintveld,

Organizations: Czech Technical University in Prague (CTU) · University of Texas at El Paso (UTEP) · Universidad Nacional de Cuyo (UNCUYO)

Abstract

Autonomous offensive agents often fail to transfer beyond the networks on which they are trained. We isolate a minimal but fundamental shift -- unseen host/subnet IP reassignment in an otherwise fixed enterprise scenario -- and evaluate attacker generalization in the NetSecGame environment. Agents are trained on five IP-range variants and tested on a sixth unseen variant; only the meta-learning agent may adapt at test time. We compare three agent families (traditional RL, adaptation agents, and LLM-based agents) and use action-distribution-based behavioral/XAI analyses to localize failure modes. Some adaptation methods show partial transfer but significant degradation under unseen reassignment, indicating that even address-space changes can break long-horizon attack policies. Under our evaluation protocol and agent-specific assumptions, prompt-driven pretrained LLM agents achieve the highest success on the held-out reassignment, but at the cost of increased inference-time compute, reduced transparency, and practical failure modes such as repetition/invalid-action loops.

Figures & tables

Figure 1: DARLA [ 12 ] intuition for generalization: A robot trained in one visual domain (e.g., a red room with a red fruit) can fail under appearance shifts (e.g., a blue room with an orange fruit) if it overfits to surface cues.

Paper	Environment	Hosts	TR	UT	Performance	Num. res.
Entity [ 32 ]	Yawning Titan sim.	n/r	NT/ep.	✓	Entity Transformer $>$ MLP baseline	$\times$
Graph [ 17 ]	Graph-based ACD	3–10	FT $\rightarrow$ NT	✓	$\sim$ 4 $\times$ baseline score	✓
Relat. [ 14 ]	Relational SysAdmin-style env.	var.	Size scale	✓	Zero-shot transfer reported	$\times$
ACD-G [ 6 ]	ACD-G custom env.	8–16	Multi-ep./topo	✓	Generalization improves over baseline	$\times$
ICS [ 20 ]	ICS testbed	$\sim$ 10	Fixed	(✓)	Autonomous mitigation demonstrated	✓
Attack [ 26 ]	Simulated attack env.	n/r	Fixed	$\times$	78% success rate	✓

Table 1: Summary of environments, training regimes (TR), unseen topology tests (UT), performance, and numerical results (Num. res.) in recent work on autonomous cybersecurity agents. TR: NT/ep.=new topology each episode, FT

\rightarrow

NT=fixed then new, Size scale=trained on smaller networks then larger, Multi-ep./topo=many episodes per topology, Fixed=no change. UT: ✓=yes,

\times

=no, (✓)=partial. Num. res.: ✓=numeric results reported,

\times

=no. n/r=not reported, var.=variable.

Action type	Additional parameters
ScanNetwork	source_host, $\text{target\_network}\in\texttt{known\_networks}$
FindServices	source_host, $\text{target\_host}\in\texttt{known\_hosts}$
FindData	source_host, $\text{target\_host}\in\texttt{known\_hosts}$
ExploitService	source_host, $\text{target\_host}\in\texttt{known\_hosts}$ , $\text{target\_service}\in\texttt{known\_services}$
ExfiltrateData	source_host, $\text{target\_host}\in\texttt{controlled\_hosts}$ , $\text{data}\in\texttt{known\_data}$
BlockIP (defender)	source_host, $\text{target\_host}\in\texttt{controlled\_hosts}$ , $\text{blocked\_host}\in\texttt{known\_hosts}$

Table 2: List of NSG action types and their parameters. The attacker agents in this paper use the first five action types; BlockIP is a defender action.

Source	Destination
Client network	Client network
Server network	Server network
Client network	smb_server
Client network	db_server
Client network	web_server
Client network	Internet

Table 3: Connections explicitly allowed in the Scenario of NSG.

Figure 2: The network topology used in the Data exfiltration scenario. There are two local sub-networks, each consisting of 5 hosts. Hosts are reachable from each other within the same sub-networks. Access from the other sub-network is determined by firewall rules (shown as dotted lines).

Panel of Figure 3 — Figure 3: DDQN agent architecture: JSON observations are embedded, mapped to a Q-value vector over the full action set, and the greedy action is selected after masking to currently valid actions (with a delayed target network for stability).

Figure 4: Dataset distribution across topologies. Win values represent successful episodes, Progress actions are state changing valid actions, and Winning Progress actions are progress actions from successful episodes.

Figure 5: Diagram of Operation of the Conceptual Agent. The translations from state to concept-state and from concept-action to action are done as a wrapper. The agent operates on the new state without information about the original state.

Concrete element	Condition	Concept representation
Host $h$	ext.	external_k
Host $h$	int. & ctrl.	host_k
Host $h$	int. & $\neg$ ctrl.	unknown_k
Host $h$ + $S(h)$	svc. discovered	host_k_p_1_p_2… (service-derived tags $p_{i}$ )
Data $d$ on host $h$	$d$ discovered	attach $d$ to host concept $c_{h}$ (i.e., $\tilde{D}(c_{h})\ni d$ )
Network $n$	int.; count $H(n)$	net_k_ $H(n)$ hosts

Table 4: Conceptual Agent: heuristic mapping from concrete state elements to address-independent concepts. Abbreviations: ext.=external, int.=internal, ctrl.=controlled, svc.=non-local service,

H(n)

=#known hosts in network

n

Conceptual action	Concept-level parameters	Translation to concrete action
Exploit service	Source host concept $\tilde{h}_{\mathrm{src}}$ , target host concept $\tilde{h}_{\mathrm{tgt}}$ , service descriptor $\tilde{s}$	$h_{\mathrm{src}}=\psi_{H}(\tilde{h}_{\mathrm{src}})$ chosen among controlled hosts, $h_{\mathrm{tgt}}=\psi_{H}(\tilde{h}_{\mathrm{tgt}})$ among known hosts; $\tilde{s}$ is kept unchanged, yielding a concrete service to exploit $(h_{\mathrm{src}},h_{\mathrm{tgt}},\tilde{s})$ .
Find services	Source concept $\tilde{h}_{\mathrm{src}}$ , target concept $\tilde{h}_{\mathrm{tgt}}$	$\psi_{H}$ is applied to obtain a concrete controlled source $h_{\mathrm{src}}$ and a concrete known target $h_{\mathrm{tgt}}$ , instantiating a service-discovery scan from $h_{\mathrm{src}}$ to $h_{\mathrm{tgt}}$ .
Find data	Source concept $\tilde{h}_{\mathrm{src}}$ , target concept $\tilde{h}_{\mathrm{tgt}}$ (both representing controlled hosts)	Both concepts are mapped via $\psi_{H}$ restricted to controlled hosts, producing concrete controlled machines $(h_{\mathrm{src}},h_{\mathrm{tgt}})$ and a data-search action $h_{\mathrm{src}}\to h_{\mathrm{tgt}}$ .
Exfiltrate data	Source concept $\tilde{h}_{\mathrm{src}}$ , target concept $\tilde{h}_{\mathrm{tgt}}$ , data item $\tilde{d}$ (all on controlled hosts)	$\psi_{H}$ yields concrete controlled hosts $h_{\mathrm{src}}$ and $h_{\mathrm{tgt}}$ ; the data identifier $\tilde{d}$ is preserved, resulting in a concrete exfiltration $(h_{\mathrm{src}},h_{\mathrm{tgt}},\tilde{d})$ .
Scan network	Source host concept $\tilde{h}_{\mathrm{src}}$ , network concept $\tilde{n}$	$h_{\mathrm{src}}=\psi_{H}(\tilde{h}_{\mathrm{src}})$ is a concrete controlled host; the target network is instantiated as $n=\psi_{N}(\tilde{n})$ , choosing one concrete network consistent with the concept, and a network scan $h_{\mathrm{src}}\to n$ is issued.

Table 5: Conceptual Agent: Heuristic translation of conceptual actions into concrete environment actions via the inverse abstractions

\psi_{H}

(hosts) and

\psi_{N}

(networks).

Filter	Description
Internal-only + firewall-aware	Keep only internal actions; drop actions that traverse a known blocked link.
No repeats (same concept-action)	Do not propose an action if the same (source, target, type) concept-action was already attempted.
Scan only when svc. unknown	Propose service-discovery only for hosts with unknown services.
No exploit on ctrl./self/local	Exploit only uncontrolled targets; disallow self-target; ignore local-only services.
FindData only if needed	Search for data only on controlled internal hosts without known data.
Exfiltrate only new relevant data	Exfiltrate only non-duplicate non-log data to another controlled host (no self-exfiltration).

Table 6: Conceptual Agent: main filters for reducing the candidate action set while preserving informative options for learning.

Agent	Key idea	State representation	Training	Knows $\|\mathcal{A}\|$ ?	Biases
Random	Random valid-action baseline	None; samples uniformly from valid actions	None	No	None
DQN	Value-based RL (no adaptation)	Candidate-action matrix with 12-D features from valid actions and partial graph	Fixed episodes	No	Feature design; action masking
MAML	Meta-learning for fast adaptation	Same candidate-action features as DQN	Meta-training (adaptation allowed)	No	Feature design; action masking
Reptile	First-order meta-learning	Same candidate-action features as DQN	Meta-training (adaptation allowed)	No	Feature design; action masking
DDQN	LM-based action scoring	JSON observation embedded by pretrained LM; Q-network scores state–action pairs	Train until convergence	Yes	Reward shaping; action masking
Conceptual	Role-based abstraction learning	Symbolic concept-state abstraction; Q-learning over concepts	Train until convergence	No	Concept abstraction; action pruning

Table 7: Summary of agent differences in representation, training, action-space knowledge, and inductive biases.

Agent	Pretrain	TT learn	Heuristics	Prune	Cost
Random	–	No	–	No	Low
DQN	–	No	Feat.	Yes	Low
DDQN	Encoder	No	Rwd.	No	Med
Conceptual	–	No	Rules + Rwd.	Yes	Low–Med
MAML	–	Yes	Feat.	Yes	Med–High
Reptile	–	Yes	Feat.	Yes	Med

Table 8: Comparability table: evaluation privileges and resource assumptions across agent families. Pretrain indicates whether the agent relies on external pretrained models (beyond randomly initialized networks trained purely in NetSecGame). TT learn indicates whether parameters are updated on

T_{unseen}

before evaluation. Heuristics summarizes hand-designed components (Feat.=feature engineering; Rules=rule-based abstractions/pruning; Rwd.=reward shaping; Prompt=prompt scaffolding/memory). Cost is a qualitative inference-time profile.

Exp #	Final Training WR	Final Evaluation WR
1	$37.18\pm 0.91$	$2.07\pm 2.35$
2	$47.60\pm 4.98$	$25.20\pm 9.81$
3	$34.55\pm 2.78$	$2.47\pm 3.07$
4	$51.53\pm 5.50$	$35.40\pm 1.73$

Table 9: Average Win Rate for DQN Experiments

Figure 6: UMAP projection of state embeddings for the same scenario under four different IP ranges.

Metric	All Episodes	Wins	Losses
Count	263	250	13
Win Rate (%)	95.1	—	—
Avg Steps	$31.2\pm 23.2$	$27.8\pm 18.2$	$96.7\pm 2.4$
Avg Total Reward	$+63.9\pm 40.9$	$+72.2\pm 18.2$	$-97.5\pm 4.3$
Min Steps	8	8	91
Max Steps	100	91	100

Table 10: Episode Summary Statistics ReAct Agent using GPT-OSS-120b on the Unseen Topology

Figure 7: Win rate performance of the ReAct agent.

Overall
Metric	Seen (T5)	Unseen
Win Rate (%)	52.2	51.6
Return	$+12.5\pm 81.6$	$-6.5\pm 81.7$
Steps	$39.7\pm 36.7$	$57.7\pm 33.6$
Wins
Return	$+87.1\pm 6.8$	$+70.8\pm 14.3$

Table 11: LLM-BERT Performance Comparison: Seen vs. Unseen Topology

Loss Type	Count	% of Losses	Avg Steps	Avg Reward
Invalid Actions Exhausted	12	92.3	96.4	-96.42
Step Limit Reached	1	7.7	100.0	-110.00
Total	13	100.0	96.8	-97.54

Table 12: Loss Type Analysis for ReAct using GPT-OSS-120b on the Unseen Network Topology

Panel of Figure 8 — Figure 8: Repetition behavior analysis: distribution of repetition rates across successful and failed episodes for the ReAct and LLM-BERT agents.

Figure 9: Conceptual Agent: Performance of each of the five training runs. Runs 0, 1, 2, and 4 stopped after reaching the threshold of 0.95 evaluation win rate. Run number 3 stopped after reaching 230,000 episodes without reaching the 0.95 evaluation win rate.

Figure 10: Conceptual Agent: Performance of each of the five training runs during training episodes. The win rate is computed while the model is still learning and changing.

Loss Type	Count (%)	Steps	Reward
Negative Return	110 (90.9)	86.5	-89.93
Step Limit	11 (9.1)	100.0	-85.25
Total	121 (100)	88.0	-89.25

Table 13: Loss Type Analysis for LLM-BERT on Unseen Network Topology

Metric	Average	Std. Dev.
Win Rate (%)	65.530	4.770
Returns	620.830	519.884
Episode Steps	67.158	30.260
Win Steps	49.894	23.209

Table 14: Conceptual agent performance in unseen scenario 6 over 100 seeds.

Panel of Figure 11 — Figure 11: Training win rate comparison of meta-learning agents.

Agent	Win rate (%)	Episode return	Episode steps	Total training episodes	# of steps for win episode
Random agent (baseline)	$6.0\pm 2.4$	$-100.93\pm 34.95$	98.47	–	–
Single Buffer DQN	$2.07\pm 2.36$	$-106.71\pm 3.69$	$98.98\pm 1.11$	5000	$10.33\pm 1.79$
Dual Buffer DQN	$3.07\pm 2.72$	$-104.45\pm 5.01$	$97.82\pm 2.02$	5000	$15.33\pm 13.59$
DDQN+emb	0.0	-99.00	100.00	–	–
ReAct (gpt-oss-120b)	$95\pm 22$	$63.86\pm 40.85$	$31.16\pm 23.21$	–	$27.76\pm 18.20$
LLM-BERT	$51.60\pm 3.16$	$-6.50\pm 81.70$	$57.66\pm 33.57$	–	$29.16\pm 14.29$

Table 15: Comparison of different generalization approaches and baseline agents in the testing (previously unseen) topology. The episode timeout is 100 steps, and the maximum possible return is 95.

Panel of Figure 12 — Figure 12: Random baseline: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 13 — Figure 13: DQN: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 14 — Figure 14: DDQN: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 15 — Figure 15: ReAct: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 16 — Figure 16: LLM-BERT: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 17 — Figure 17: Conceptual Q-learning: action-type distribution over time on seen vs. unseen topology.

Panel of Figure 18 — Figure 18: MAML: action-type distribution over time on seen vs. unseen topology (after test-time adaptation).

Panel of Figure 19 — Figure 19: Reptile: action-type distribution over time on seen vs. unseen topology (after test-time adaptation).

Generalization in RL

Reinforcement learning generalization studies how agents transfer learned behavior to unseen tasks, environments, dynamics, and observations.

See all 89 papers

May 8, 2026

CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, each agent exhibits an attack-selection bias, disproportionately concentrating its efforts on a narrow subset of attack families regardless of prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evaluates five agents on three targets and four prompt conditions with ten attack families. We identify explicit bias across agents, with different dominant attack families and varying entropy levels in their attack-family allocation distributions. Such bias is better characterized as a trait of the agents, rather than a factor associated with the attack success rate. Furthermore, our experiments reveal a bias momentum effect, where agents resist explicit steering toward attack families that conflict with their bias. This forced distribution shift does not yield measurable improvements in attack performance. To ensure reproducibility and facilitate future research, we release an interactive result dashboard at https://trustworthyai.co.kr/CyBiasBench/ and a reproducibility artifact with aggregated session-level statistics and full evaluation scripts at https://github.com/Harry24k/CyBiasBench.

Taein Lim, Seongyong Ju, Munhyeok Kim +2

Chung-Ang University · Myongji University

AI Agent Security Benchmarks LLMs for Cybersecurity Language Model Bias Evaluation

Ask PDF

Mar 12, 2026

Can RL Improve Generalization of LLM Agents? An Empirical Study

Reinforcement fine-tuning (RFT) has shown promise for training LLM agents to perform multi-turn decision-making based on environment feedback. However, most existing evaluations remain largely in-domain: training and testing are conducted in the same environment or even on the same tasks. In real-world deployment, agents may operate in unseen environments with different background knowledge, observation spaces, and action interfaces. To characterize the generalization profile of RFT under such shifts, we conduct a systematic study along three axes: (1) within-environment generalization across task difficulty, (2) cross-environment transfer to unseen environments, and (3) sequential multi-environment training to quantify transfer and forgetting. Our results show that RFT generalizes well across task difficulty within an environment, but exhibits weaker transfer to unseen environments, which correlates with shifts in both semantic priors and observation/action interfaces. In contrast, sequential training yields promising downstream gains with minimal upstream forgetting, and mixture training across environments improves the overall balance. We further provide detailed analyses and deeper insights, and hope our work helps the community develop and deploy generalizable LLM agents.

Zhiheng Xi, Xin Guo, Jiaqi Liu +11

Fudan University · Shanghai Artificial Intelligence Laboratory · Meituan

LLM Agents RL for LLM Agents RL Fine-Tuning Generalization in RL Continual Learning for LLM Agents Multi-Task RL

Ask PDF

Apr 29, 2026

Autonomous LLM Agents & CTFs: A Second Look

Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs.

Youness Bouchari, Matteo Boffa, Marco Mellia +3

Politecnico di Torino · Universit`a di Torino · Huawei Technologies France

LLM Agent Orchestration AI Agent Security Benchmarks LLMs for Cybersecurity LLM Agent Reliability

Ask PDF

Abstract

Generalization in RL

Explore similar work

CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Can RL Improve Generalization of LLM Agents? An Empirical Study

Autonomous LLM Agents & CTFs: A Second Look