Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through verified feedback loops. Yet, scalable and generalizable execution-grounded environments remain scarce, limiting progress in training more capable ML agents. We introduce CTF-Dojo, the first large-scale executable runtime tailored for training LLMs with verifiable feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style challenges containerized in Docker with guaranteed reproducibility. To enable rapid scaling without manual intervention, we develop CTF-Forge, an automated pipeline that transforms publicly available artifacts into ready-to-use execution environments in minutes, eliminating weeks of expert configuration traditionally required. We trained LLM-based agents on just 486 high-quality, execution-verified trajectories from CTF-Dojo, achieving up to 9.9% absolute gains over strong baselines across three competitive benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1, establishing a new open-weight state-of-the-art that rivals frontier models like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a benchmark for executable-agent learning, CTF-Dojo demonstrates that execution-grounded training signals are not only effective but pivotal in advancing high-performance ML agents without dependence on costly proprietary systems.
Figures & tables
Figure 1: CTF-Forge powers automated creation of configuration files from publicly sourced CTF artifacts for containerizing CTF challenges.
Executable Environment
Detection
Exploitation
Agentic
Real Task
# Total
# Train
SecRepoBench ( Dilgren et al., 2025 )
✗
✗
✓
✓
318
0
CVE-Bench ( Wang et al., 2025a )
✗
✗
✓
✓
509
0
CVE-Bench ( Zhu et al., 2025 )
✗
✓
✓
✓
509
0
SEC-bench ( Lee et al., 2025 )
✗
✓
✓
✓
1,507
0
CyberGym ( Wang et al., 2025b )
✗
✓
✓
✓
1,507
0
CyberSecEval 3 ( Wan et al., 2024 )
✓
✓
✓
✗
6
0
Table 1: CTF-Dojo is the first cybersecurity executable environment deriving agent trajectories for training. Detection: whether the task requires vulnerability detection; exploitation: whether the task needs LLMs to verify the detected vulnerabilities; Agentic: whether each instance is repaired with an interactive environment for exploitation; Real Task: whether each instance is developed by human experts.
Benchmark
Level
# Competition
# Crypto
# Forensics
# Pwn
# Rev
# Web
# Misc
# Total
Training
CTF-Dojo
Multi-Level
50
228
38
163
123
21
85
658
Evaluation
InterCode-CTF
High School
1
16
13
2
27
2
31
91
NYU CTF Bench
University
1
51
14
38
51
14
24
192
Cybench
Professional
4
16
4
2
6
8
4
40
Table 2: Challenge distribution across CTF datasets.
Figure 2: Solved challenges.
Model
Train Size
InterCode-CTF
NYU CTF
Cybench
Average
Proprietary Models
Claude-3.7-Sonnet ( Anthropic, 2025a )
-
86.8
18.2
30.0
39.0
Claude-3.5-Sonnet ( Anthropic, 2024 )
-
85.7
16.7
25.0
37.2
Gemini-2.5-Flash ( Comanici et al., 2025 )
-
81.3
14.1
17.5
33.4
Open Weight Models
DeepSeek-V3-0324 ( Liu et al., 2024 )
-
82.5
6.2
27.5
30.3
Table 3: Pass@1 performance on benchmark tasks. The improvements of CTF-Dojo are absolute in comparison with the Qwen3 model of corresponding sizes.
Figure 3: Effect of data scaling. Models across sizes benefit from increased number of training trajectories.
Models
# Crypto
# Forensics
# Pwn
# Rev
# Web
# Misc
# Total
–
+
–
+
–
+
–
+
–
+
–
+
–
+
Proprietary Models
Claude-3.7-Sonnet
41.2
50.9
42.1
50.0
14.7
20.9
41.5
49.6
61.9
76.2
47.1
69.4
36.2
46.4
Claude-3.5-Sonnet
39.9
43.9
39.5
47.4
8.0
13.5
39.8
41.5
47.6
57.1
45.9
68.2
33.0
39.7
Open Weight Models
DeepSeek-V3-0324
37.1
41.0
41.0
43.6
12.0
13.5
34.1
36.6
33.3
52.4
36.5
41.2
30.4
33.9
Table 4: Solved rate (%) on CTF-Dojo tasks across categories, using EnIGMA+ . “–” indicates baseline without writeup hints; “+” includes writeups in the prompt.
Figure 4: Effect of runtime augmentation.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Item Description
Count
CTF-Dojo Challenges
Number of available CTF challenges
658
Number of challenges with stable and reproducible environments, as confirmed by the original authors
650
Writeups for CTF Challenges
Total number of writeups collected from the CTFtime website
8,361
Writeups successfully matched to CTF-Dojo challenges using competition and task metadata
252
Appendix
Table 5: Summary of data statistics.
Figure 5: Number of turns in each successful trajectory (left) and number of successful trajectories for each challenge instance (right).
Category
Qwen
Both
DeepSeek
Crypto
31
84
26
Forensics
1
13
3
Pwn
2
15
3
Rev
6
37
9
Web
0
6
2
Misc
4
26
6
Appendix
Table 6: Solved challenge counts.
Teacher Model
8B
14B
32B
Qwen3-Coder
17.3
23.8
29.4
DeepSeek-V3-0324
17.6
24.8
31.3
Combined
18.9
25.7
31.9
Appendix
Table 7: Pass@1 performance when varying teacher models.
Disassembles a binary and prints the disassembly of a given function name, or main by default.
Dynamic analysis
debug_start <binary> [<args>]
Starts a debug session with the given binary and optional command-line arguments for the binary.
debug_add_breakpoint <breakpoint>
Adds a breakpoint in the debug session.
debug_continue
Continues the program execution in the debug session.
debug_step [<number>]
Steps number of instructions (default is 1) in the debug session.
Appendix
Table 9: In additional to the standard Linux Bash commands and the SWE-agent specialized tools, we provide EnIGMA with tools in the offensive cybersecurity domain, including binary decompilation and disassemble, and interactive agent tools for debugging and connecting to remote servers. Required arguments are enclosed in <> and optional arguments are in [] . The last column shows the documentation presented to the LLMs.