How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation
Authors: Kimberly Milner, Minghao Shao, Nanda Rani, Haoran Xi, Venkata Sai Charan Putrevu, Meet Udeshi, Sandeep K. Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Muhammad Shafique, Ramesh Karri
Organizations: NYU Tandon School of Engineering · NYU Abu Dhabi · CISPA - Helmholtz Center for Information Security, Saarbrücken · Indian Institute of Technology Tirupati · International Institute of Information Technology Hyderabad
Capture-the-Flag (CTF) benchmarks are widely used to assess the offensive security capabilities of autonomous language-model agents. Evaluations rely on shallow binary judgments or aggregate scores, overlooking the agent's trajectory to the flag. Consequently actual exploitation is conflated with direct flag exposure, memorized recall, external lookup, guessing, and unsupported claims, potentially overstating the agent's cybersecurity capability. We introduce CTF-ABACUS, a trace-based agent auditing framework that reconstructs each run as an evidence-grounded solve profile. By decomposing agent actions into penetration-testing phases and categorical techniques, it identifies where exploitation occurs, where the flag first appears, and whether the recovered flag is supported by demonstrated behavior. Aggregating solve profiles across agents yields challenge signatures that reveal whether success was achieved via the intended exploit or via shortcut pathways. We apply CTF-ABACUS to 1,435 CTF attempts by six frontier and open-source models on 240 challenges, yielding 2,870 solve profiles under two judge lenses. Trace-verified exploits account for only 62-87% of recovered flags across benchmarks, while shortcut recoveries follow substantially shallower trajectories. These findings shift CTF evaluation from counting recovered flags to verifying demonstrated exploitation and provide a basis for designing benchmarks that better isolate the offensive capabilities.
Figures & tables
Figure 1: CTF-Abacus reconstructs each run from its execution trace, separating demonstrated exploitation from shortcuts that flag matching counts as solved.
Item
Symbol
Meaning
Trace notation
Execution trace
T=(e1,…,en)
Ordered sequence of n execution steps.
Step
ei=(ai,oi)
Agent action ai and corresponding observation oi .
Step labels
zi=(ϕi,τi)
PTES phase ϕi and standards-grounded technique τi .
Table 1: Solve-profile notation and flag-provenance rules.
Figure 2: From execution trace to verified solve profile. Multi-agent trajectories are linearized into steps (ai,oi) , labeled with penetration-testing phases ϕi and techniques τi , assigned flag provenance π , evidence-audited, and aggregated into challenge signatures. Human review is restricted to reasoning-originated flags classified as recalled, guessed, leaked, or derived .
Figure 3: Trajectory geometry across category, benchmark, and model, under both judge lenses. Diagonal, zigzag, and baseline composition of all evaluable runs, cut three ways. Each group shows the Claude lens (C, upper bar) over the GPT lens (G, lower bar). Composition shifts sharply across category and benchmark but barely across model; the two lenses track closely.
Phase
Steps / # τ ( Eff )
Dominant technique (share)
Reconnaissance
52% / 71 (12)
Probe live service (27%)
Vulnerability analysis
19% / 126 (42)
Category-specific ( ≤ 18%)
Exploitation
15% / 66 (13)
Author solve script (40%)
Post-exploitation
< 2% / 12 (2)
Read captured flag (80%)
Table 2: Effort and technique diversity by PTES ϕ
# τ
PTES depth
monotonicity m
reached Exp.
Genuine
5.2
4.3
0.80
89%
Shortcut
4.3
2.8
0.50
42%
Table 3: genuine & ¬ϕ =Exploit Flag recoveries.
τ
m
d
gen.
r.Exp
rec.
r.Post
CTFTiny
6.5
0.75
4.21
86
88
86
40
Cybench
9.0
0.50
3.58
62
70
36
16
HTB
5.5
0.79
4.13
81
91
87
22
Intercode
2.5
0.65
3.60
64
64
79
23
Table 4: Benchmark quality by dimension , median over 6 models. τ techniques; m monotonicity (0–1); d PTES depth (0–5); gen. genuine share of recoveries; r.Exp / rec. / r.Post % of runs reaching Exploitation / recovering / reaching Post-Exploitation
Figure 5: Command- and run-summary features poorly distinguish execution-backed from unsupported flag recoveries.
Figure 6: Verification shifts model scores similarly, while the adjustment varies more across benchmarks, making rank changes primarily benchmark-dependent.
Figure 7: Recovery provenance varies substantially across benchmarks, with direct flag exposure accounting for far more unsupported recoveries than contamination.
Figure 8: Solve pathways remain mixed across models even after crediting verified derivations, while identical terminal scores can represent different evidence.
Figure 9: Execution-backed and unsupported recoveries follow coherent but opposing behavioral patterns, although no single feature reliably identifies an individual pathway.
Figure 10: Trajectories with no demonstrated exploitation
Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs.
Youness Bouchari, Matteo Boffa, Marco Mellia +3
Politecnico di Torino · Universit`a di Torino · Huawei Technologies France
Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We present DeepRed, an open-source benchmark for evaluating LLM-based agents on realistic Capture The Flag (CTF) challenges in isolated virtualized environments. DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis. To move beyond binary solved/unsolved outcomes, we introduce a partial-credit scoring method based on challenge-specific checkpoints derived from public writeups, together with an automated summarise-then-judge labelling pipeline for assigning checkpoint completion from logs. Using DeepRed, we benchmark ten commercially accessible LLMs on ten VM-based CTF challenges spanning different challenge categories. The results indicate that current agents remain limited: the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery and longer-horizon adaptation.
Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3
Delft University of Technology Delft, The Netherlands
Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent application. To evaluate such agents, researchers widely adopt Capture The Flag (CTF) benchmarks. However, current CTF benchmarks reuse existing challenges, which exposes them to data contamination and potential cheating. Notably, we confirmed these issues in practice by integrating web search tools into an existing agent. To address these limitations, we present CTFusion, a streaming evaluation framework built on Live CTFs. To achieve this, CTFusion preserves per-agent independence under a single team account and reduces competition impact by forwarding only the first correct flag per challenge. Moreover, we implement CTFusion as a Model Context Protocol (MCP) server on the widely used CTFd platform, which offers broad applicability to diverse CTF events and agent types. Through experiments with three LLMs, two agents, and five Live CTFs, we demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents. We release CTFusion as open source to foster future research in this area.
Dongjun Lee, Ga-eun Bae, Insu Yun
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea