cs.SEAug 25, 2025

Training Language Model Agents to Find Vulnerabilities with CTF-Dojo

Authors: Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, Zijian Wang

Organizations: Monash University · AWS AI Labs

Abstract

Large language models (LLMs) have demonstrated exceptional capabilities when trained within executable runtime environments, notably excelling at software engineering tasks through verified feedback loops. Yet, scalable and generalizable execution-grounded environments remain scarce, limiting progress in training more capable ML agents. We introduce CTF-Dojo, the first large-scale executable runtime tailored for training LLMs with verifiable feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style challenges containerized in Docker with guaranteed reproducibility. To enable rapid scaling without manual intervention, we develop CTF-Forge, an automated pipeline that transforms publicly available artifacts into ready-to-use execution environments in minutes, eliminating weeks of expert configuration traditionally required. We trained LLM-based agents on just 486 high-quality, execution-verified trajectories from CTF-Dojo, achieving up to 9.9% absolute gains over strong baselines across three competitive benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1, establishing a new open-weight state-of-the-art that rivals frontier models like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a benchmark for executable-agent learning, CTF-Dojo demonstrates that execution-grounded training signals are not only effective but pivotal in advancing high-performance ML agents without dependence on costly proprietary systems.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Autonomous LLM Agents & CTFs: A Second Look

    Apr 29, 2026Youness Bouchari, Matteo Boffa, Marco Mellia +3LLM Agent OrchestrationAI Agent Security Benchmarks

  2. CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

    May 12, 2026Dongjun Lee, Ga-eun Bae, Insu YunBenchmark ContaminationLLM Agent Evaluation

  3. Cyber-Zero: Training Cybersecurity Agents without Runtime

    Jul 29, 2025Terry Yue Zhuo, Dingmin Wang, Hantian Ding +2LLMs for CybersecurityLLM Agent Training