How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation

Kimberly Milner; Minghao Shao; Nanda Rani; Haoran Xi; Venkata Sai Charan Putrevu; Meet Udeshi; Sandeep K. Shukla; Prashanth Krishnamurthy

cs.CRAug 26, 2026

How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation

Authors: Kimberly Milner, Minghao Shao, Nanda Rani, Haoran Xi, Venkata Sai Charan Putrevu, Meet Udeshi, Sandeep K. Shukla, Prashanth Krishnamurthy,

Organizations: NYU Tandon School of Engineering · NYU Abu Dhabi · CISPA - Helmholtz Center for Information Security, Saarbrücken · Indian Institute of Technology Tirupati ·

Abstract

Capture-the-Flag (CTF) benchmarks are widely used to assess the offensive security capabilities of autonomous language-model agents. Evaluations rely on shallow binary judgments or aggregate scores, overlooking the agent's trajectory to the flag. Consequently actual exploitation is conflated with direct flag exposure, memorized recall, external lookup, guessing, and unsupported claims, potentially overstating the agent's cybersecurity capability. We introduce CTF-ABACUS, a trace-based agent auditing framework that reconstructs each run as an evidence-grounded solve profile. By decomposing agent actions into penetration-testing phases and categorical techniques, it identifies where exploitation occurs, where the flag first appears, and whether the recovered flag is supported by demonstrated behavior. Aggregating solve profiles across agents yields challenge signatures that reveal whether success was achieved via the intended exploit or via shortcut pathways. We apply CTF-ABACUS to 1,435 CTF attempts by six frontier and open-source models on 240 challenges, yielding 2,870 solve profiles under two judge lenses. Trace-verified exploits account for only 62-87% of recovered flags across benchmarks, while shortcut recoveries follow substantially shallower trajectories. These findings shift CTF evaluation from counting recovered flags to verifying demonstrated exploitation and provide a basis for designing benchmarks that better isolate the offensive capabilities.

Figures & tables

Figure 1: CTF-Abacus reconstructs each run from its execution trace, separating demonstrated exploitation from shortcuts that flag matching counts as solved.

Item	Symbol	Meaning
Trace notation
Execution trace	$T=(e_{1},\ldots,e_{n})$	Ordered sequence of $n$ execution steps.
Step	$e_{i}=(a_{i},o_{i})$	Agent action $a_{i}$ and corresponding observation $o_{i}$ .
Step labels	$z_{i}=(\phi_{i},\tau_{i})$	PTES phase $\phi_{i}$ and standards-grounded technique $\tau_{i}$ .
PTES depth
Phase order	$1\rightarrow 5$	Pre-engagement $\rightarrow$ Intelligence Gathering $\rightarrow$ Vulnerability Analysis $\rightarrow$ Exploitation $\rightarrow$ Post-Exploitation.

Table 1: Solve-profile notation and flag-provenance rules.

Figure 2: From execution trace to verified solve profile. Multi-agent trajectories are linearized into steps

(a_{i},o_{i})

, labeled with penetration-testing phases

\phi_{i}

and techniques

\tau_{i}

, assigned flag provenance

\pi

, evidence-audited, and aggregated into challenge signatures. Human review is restricted to reasoning-originated flags classified as recalled, guessed, leaked, or derived .

Figure 3: Trajectory geometry across category, benchmark, and model, under both judge lenses. Diagonal, zigzag, and baseline composition of all evaluable runs, cut three ways. Each group shows the Claude lens (C, upper bar) over the GPT lens (G, lower bar). Composition shifts sharply across category and benchmark but barely across model; the two lenses track closely.

Phase	Steps / # $\tau$ ( Eff )	Dominant technique (share)
Reconnaissance	52% / 71 (12)	Probe live service (27%)
Vulnerability analysis	19% / 126 (42)	Category-specific ( $\leq$ 18%)
Exploitation	15% / 66 (13)	Author solve script (40%)
Post-exploitation	$<$ 2% / 12 (2)	Read captured flag (80%)

Table 2: Effort and technique diversity by PTES

\phi

	# $\tau$	PTES depth	monotonicity $m$	reached Exp.
Genuine	5.2	4.3	0.80	89%
Shortcut	4.3	2.8	0.50	42%

Table 3: genuine &

\neg\phi

=Exploit Flag recoveries.

	$\tau$	$m$	$d$	gen.	r.Exp	rec.	r.Post
CTFTiny	6.5	0.75	4.21	86	88	86	40
Cybench	9.0	0.50	3.58	62	70	36	16
HTB	5.5	0.79	4.13	81	91	87	22
Intercode	2.5	0.65	3.60	64	64	79	23

Table 4: Benchmark quality by dimension , median over 6 models.

\tau

techniques;

m

monotonicity (0–1);

d

PTES depth (0–5); gen. genuine share of recoveries; r.Exp / rec. / r.Post % of runs reaching Exploitation / recovering / reaching Post-Exploitation

Figure 4: Benchmark decomposition Axes: $\tau$ techniques; (mono) monotonocity rate, ( depth ) (PTES); ( gen ) genuine solve share; ( rExp ) reach Exploitation; ( rec ) recovery, ( rPost ) reach Post-Exploitation. Color bands shows per-model min-max; no clipping.

Figure 5: Command- and run-summary features poorly distinguish execution-backed from unsupported flag recoveries.

Figure 6: Verification shifts model scores similarly, while the adjustment varies more across benchmarks, making rank changes primarily benchmark-dependent.

Figure 7: Recovery provenance varies substantially across benchmarks, with direct flag exposure accounting for far more unsupported recoveries than contamination.

Figure 8: Solve pathways remain mixed across models even after crediting verified derivations, while identical terminal scores can represent different evidence.

Figure 9: Execution-backed and unsupported recoveries follow coherent but opposing behavioral patterns, although no single feature reliably identifies an individual pathway.

Figure 10: Trajectories with no demonstrated exploitation

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 305 papers

Autonomous LLM Agents & CTFs: A Second Look
Apr 29, 2026Youness Bouchari, Matteo Boffa, Marco Mellia +3LLM Agent Orchestration AI Agent Security Benchmarks
Politecnico di Torino · Universit`a di Torino · Huawei Technologies France
Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges
Apr 21, 2026Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3LLM Agent Evaluation AI Agent Security Benchmarks
Delft University of Technology Delft, The Netherlands
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
May 12, 2026Dongjun Lee, Ga-eun Bae, Insu YunBenchmark Contamination LLM Agent Evaluation
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea

Apr 29, 2026J/K move · Enter open · S save

Autonomous LLM Agents & CTFs: A Second Look

Youness Bouchari, Matteo Boffa, Marco Mellia, Idilio Drago+2

Politecnico di Torino · Universit`a di Torino · Huawei Technologies France

LLM Agent Orchestration AI Agent Security Benchmarks LLMs for Cybersecurity LLM Agent Reliability

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly proposed to automate offensive security tasks, with recent studies reporting near human-level success rates in Capture-the-Flag (CTF) challenges. We here revisit these results, providing a second look at these claims. We engineer different agent architectures of increasing complexity and modularity on 30 web-based CTFs challenges spanning 14 vulnerability classes. We instantiate these agents with multiple LLM backbones, and compare them with claude-code, a general-purpose agent that automatically determines its internal architecture. Our evaluation yields three main findings. First, claude-code achieves performance comparable to the engineered architectures (19/30 solved tasks), suggesting that general-purpose agents are strong baselines for offensive security tasks. Second, both our architectures and claude-code struggle in the same challenge categories, revealing persistent barriers that keep current agents below human-level capability. Third, by leveraging our manually designed architectures we can systematically measure the impact of additional components, finding that structured orchestration of specialized roles outperforms monolithic designs, improving run-to-run consistency, and reducing execution costs.

Abstract

AI Agent Security Benchmarks

Explore similar work

Autonomous LLM Agents & CTFs: A Second Look

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges

CTFusion: A CTF-based Benchmark for LLM Agent Evaluation