Cyber-Zero: Training Cybersecurity Agents without Runtime
Organizations: Monash University · AWS AI Labs
Abstract
Large Language Models (LLMs) have achieved remarkable success in software engineering tasks when trained with executable runtime environments, particularly in resolving GitHub issues. However, such runtime environments are often unavailable in other domains, especially cybersecurity, where challenge configurations and execution contexts are ephemeral or restricted. We present Cyber-Zero, the first runtime-free framework for synthesizing high-quality agent trajectories to train cybersecurity LLMs. Cyber-Zero leverages publicly available CTF writeups and employs persona-driven LLM simulation to reverse-engineer runtime behaviors and generate realistic, long-horizon interaction sequences without actual environments. Using trajectories synthesized by Cyber-Zero, we train LLM-based agents that achieve up to 11.4% absolute performance gains over baseline models on three prominent CTF benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best model, Cyber-Zero-32B, establishes new state-of-the-art performance among open-weight models, matching the capabilities of proprietary systems like DeepSeek-V3-0324 and Claude-3.5-Sonnet while offering superior cost-effectiveness, and demonstrating that runtime-free trajectory synthesis can effectively democratize the development of state-of-the-art cybersecurity agents.
Figures & tables
| Dataset | Detection | Exploitation | Agentic | Real Task | # Total | # Train |
| SecLLMHolmes ( Ullah et al., 2024 ) | ✓ | ✗ | ✗ | ✓ | 208 | 0 |
| PrimeVul ( Ding et al., 2024 ) | ✓ | ✗ | ✗ | ✓ | 6,968 | 5,574 |
| SecRepoBench ( Dilgren et al., 2025 ) | ✗ | ✗ | ✓ | ✓ | 318 | 0 |
| CVE-Bench ( Wang et al., 2025 ) | ✗ | ✗ | ✓ | ✓ | 509 | 0 |
| CyberSecEval 3 ( Wan et al., 2024 ) | ✓ | ✓ | ✓ | ✗ | 6 | 0 |
| SecCodePLT ( Yang et al., 2024b ) | ✓ | ✓ | ✓ | ✗ | 1,345 | 0 |
| Benchmark | Level | # Crypto | # Forensics | # Pwn | # Rev | # Web | # Misc | # Total |
| InterCode-CTF | High School | 16 | 13 | 2 | 27 | 2 | 31 | 91 |
| NYU CTF Bench | University | 51 | 14 | 38 | 51 | 14 | 24 | 192 |
| Cybench | Professional | 16 | 4 | 2 | 6 | 8 | 4 | 40 |
| Total | – | 83 | 31 | 42 | 84 | 24 | 59 | 323 |
| Model | InterCode-CTF (%) | NYU CTF (%) | Cybench (%) | Average (%) | ||||||||
| Size | ZS | FT | ZS | FT | ZS | FT | ZS | FT | ||||
| Qwen-3 (main) | ||||||||||||
| 8B | 41.8 | 64.8 | +23.0 | 1.6 | 6.3 | +4.7 | 5.0 | 10.0 | +5.0 | 13.3 | 23.2 | +9.9 |
| 14B | 60.4 | 73.6 | +13.2 | 2.6 | 9.9 | +7.3 | 12.5 | 20.0 | +7.5 | 20.1 | 29.1 | +9.0 |
| 32B | 65.9 | 82.4 | +16.5 | 4.7 | 13.5 | +8.8 | 5.0 | 17.5 | +12.5 | 22.0 | 33.4 | +11.4 |
| Qwen2.5-Instruct | ||||||||||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Command | Documentation |
| Static analysis | decompile <binary_path> [--function_name <function_name>] | Decompile a binary and prints the decompilation of a given function name, or main by default. |
| disassemble <binary_path> [--function_name <function_name>] | Disassembles a binary and prints the disassembly of a given function name, or main by default. | |
| Dynamic analysis | debug_start <binary> [<args>] | Starts a debug session with the given binary and optional command-line arguments for the binary. |
| debug_add_breakpoint <breakpoint> | Adds a breakpoint in the debug session. | |
| debug_continue | Continues the program execution in the debug session. | |
| debug_step [<number>] | Steps number of instructions (default is 1) in the debug session. |
| Model | Parameter (B) | Cost ($) | Pass@1 (%) |
| Zero-Shot | |||
| Claude-3.7-Sonnet | 1̃75 | 44.44 | 43.3 |
| Claude-3.5-Sonnet | 1̃75 | 22.16 | 37.2 |
| DeepSeek-V3-0324 | 671 | 2.81 | 30.3 |
| Gemini-2.5-Flash | – | 3.09 | 33.4 |
| Qwen3-32B | 32 | 0.23 | 22.0 |
| Model | InterCode-CTF ( %) | NYU CTF ( %) | Cybench ( %) | Average ( %) | ||||||||
| Size | ZS | FT | ZS | FT | ZS | FT | ZS | FT | ||||
| 8B | 36.0 | 11.1 | -24.9 | 44.0 | 21.1 | -22.9 | 51.6 | 22.9 | -28.7 | 46.4 | 19.2 | -27.2 |
| 14B | 11.1 | 7.8 | -3.3 | 27.0 | 15.4 | -11.6 | 27.8 | 10.9 | -16.9 | 22.5 | 10.6 | -11.9 |
| 32B | 6.7 | 1.1 | -5.6 | 22.9 | 15.4 | -7.5 | 22.0 | 8.8 | -13.2 | 17.4 | 7.4 | -10.0 |
| Synthesis | InterCode-CTF (%) | NYU CTF (%) | Cybench (%) | |||
| Stuck ( ) | Pass@1 ( ) | Stuck ( ) | Pass@1 ( ) | Stuck ( ) | Pass@1 ( ) | |
| Qwen3-8B | ||||||
| Zero-Shot | 36.0 | 41.8 | 44.0 | 1.6 | 51.6 | 5.0 |
| Single-Turn | 73.5 | 25.3 | 73.9 | 0.0 | 81.6 | 0.0 |
| Multi-Turn | 11.1 | 64.8 | 21.1 | 6.3 | 22.9 | 10.0 |
| Qwen3-14B | ||||||
LLMs for Cybersecurity
Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.
See all 280 papers