STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving

James Hugglestone; Samuel Jacob Chacko; Dawson Stoller; Ryan Schmidt; Xiuwen Liu

cs.CRMar 23, 2026

STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving

Authors: James Hugglestone, Samuel Jacob Chacko, Dawson Stoller, Ryan Schmidt, Xiuwen Liu

Organizations: Department of Computer Science, Florida State University, Tallahassee, USA

Abstract

Large Language Models (LLMs) have demonstrated potential in code generation, yet they struggle with the multi-step, stateful reasoning required for offensive cybersecurity operations. Existing research often relies on static benchmarks that fail to capture the dynamic nature of real-world vulnerabilities. In this work, we introduce STRIATUM-CTF (A Search-based Test-time Reasoning Inference Agent for Tactical Utility Maximization in Cybersecurity), a modular agentic framework built upon the Model Context Protocol (MCP). By standardizing tool interfaces for system introspection, decompilation, and runtime debugging, STRIATUM-CTF enables the agent to maintain a coherent context window across extended exploit trajectories. We validate this approach not merely on synthetic datasets, but in a live competitive environment. Our system participated in a university-hosted Capture-the-Flag (CTF) competition in late 2025, where it operated autonomously to identify and exploit vulnerabilities in real-time. STRIATUM-CTF secured First Place, outperforming 21 human teams and demonstrating strong adaptability in a dynamic problem-solving setting. We analyze the agent's decision-making logs to show how MCP-based tool abstraction significantly reduces hallucination compared to naive prompting strategies. These results suggest that standardized context protocols are a critical path toward robust autonomous cyber-reasoning systems.

Figures & tables

Fig. 1: High-Level Neuro-Symbolic Architecture. The system decouples probabilistic reasoning from deterministic execution. The Reasoning Layer (Left) acts as the strategic planner, emitting JSON payloads that must pass through the Protocol Layer (Center). This symbolic interface enforces strict schema validation, effectively filtering out hallucinated commands, before invoking tools in the Execution Layer (Right). The resulting system feedback (stdout/stderr) is structurally parsed and re-injected into the context window, grounding the agent’s latent state in verifiable reality.

Fig. 2: STRIATUM-CTF Execution Workflow: A sequence trace showing the transition from User Input to Flag Capture. The diagram highlights the system’s error-recovery capability: when the agent attempts an invalid tool configuration (Phase 2), the Protocol Layer enforces schema compliance, triggering an autonomous correction cycle that enables the final successful exploitation (Phase 3).

Category	Challenges
Memory Corruption	4
Reverse Engineering	4
Web Exploitation	4
Cryptography	3
Total	15

TABLE I: Benchmark dataset composition by vulnerability category.

Fig. 3: Success rates of different settings with the 95% Wilson Score confidence interval.

Condition	Success Rate	Mean Duration
Baseline	86.7% (39/45)	17.1 min
Templates	84.4% (38/45)	18.5 min
Lessons	82.2% (37/45)	19.1 min
Minimal	77.8% (35/45)	20.1 min

TABLE II: Performance summary across ablation conditions.

Fig. 4: Distribution of the time taken to solve CTF problems under different settings.

Fig. 5: Time taken of individual runs under different settings.

Fig. 6: Cumulative solves progression of the top 10 teams during the four-hour (with a 15-minute extension) competition. STRIATUM-CTF (Team 186) maintained a consistent lead throughout.

Fig. 7: Relative frequencies of different scores estimated using a kernel density estimator.

LLMs for Cybersecurity

Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.

See all 255 papers

Autonomous LLM Agents & CTFs: A Second Look
Apr 29, 2026Youness Bouchari, Matteo Boffa, Marco Mellia +3LLM Agent Orchestration AI Agent Security Benchmarks
Politecnico di Torino · Universit`a di Torino · Huawei Technologies France
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
May 12, 2026Dongjun Lee, Ga-eun Bae, Insu YunBenchmark Contamination LLM Agent Evaluation
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea
How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation
Aug 26, 2026Kimberly Milner, Minghao Shao, Nanda Rani +8AI Agent Security Benchmarks LLMs for Cybersecurity
NYU Tandon School of Engineering · NYU Abu Dhabi · CISPA - Helmholtz Center for Information Security, Saarbrücken +2

Apr 29, 2026J/K move · Enter open · S save

Autonomous LLM Agents & CTFs: A Second Look

Youness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago+2

Politecnico di Torino · Universit`a di Torino · Huawei Technologies France

LLM Agent Orchestration AI Agent Security Benchmarks LLMs for Cybersecurity LLM Agent Reliability

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs.

Abstract

LLMs for Cybersecurity

Explore similar work

Autonomous LLM Agents & CTFs: A Second Look

CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation