Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

Jiaren Peng; Zeqin Li; Chang You; Yan Wang; Hanlin Sun; Xuan Tian; Shuqiao Zhang; Junyi Liu

cs.CRApr 7, 2026

Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

Authors: Jiaren Peng, Zeqin Li, Chang You, Yan Wang, Hanlin Sun, Xuan Tian, Shuqiao Zhang, Junyi Liu,

Organizations: School of Cyber Science and Engineering, Sichuan University, China · Institute for Network Sciences and Cyberspace, Tsinghua University, China · National University of Singapore, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore ·

Abstract

The rapid advancement of Large Language Models (LLMs) has created new opportunities for Automated Penetration Testing (AutoPT), spawning numerous frameworks aimed at achieving end-to-end autonomous attacks. However, despite the proliferation of related studies, existing research generally lacks systematic architectural analysis and large-scale empirical comparisons under a unified benchmark. Therefore, this paper presents the first Systematization of Knowledge (SoK) focusing on the architectural design and comprehensive empirical evaluation of current LLM-based AutoPT frameworks. At systematization level, we comprehensively review existing framework designs across six dimensions: agent architecture, agent plan, agent memory, agent execution, external knowledge, and benchmarks. At empirical level, we conduct large-scale experiments on 13 representative open-source AutoPT frameworks and 2 baseline frameworks utilizing a unified benchmark. The experiments consumed over 10 billion tokens in total and generated more than 1,500 execution logs, which were manually reviewed and analyzed over four months by a panel of more than 15 researchers with expertise in cybersecurity. By investigating the latest progress in this rapidly developing field, we provide researchers with a structured taxonomy to understand existing LLM-based AutoPT frameworks and a large-scale empirical benchmark, along with promising directions for future research.

Figures & tables

Figure 1: The systematization framework of AutoPT. The upper section aligns with the traditional PT lifecycle. The lower section systematically deconstructs the AutoPT architecture into six core dimensions: Agent Architecture, Agent Plan, Agent Memory, Agent Execution, External Knowledge, and Benchmarks.

Framework	Planning	Execution	Summarization
ARACNE Nieponice et al. (2025)	Planner	Interpreter	Summarizer
AutoAttacker Xu et al. (2024)	Planner	Navigator	Summarizer
AutoPentest Henke (2025)	Planner	Specialised Workers
CHECKMATE Wang et al. (2025a)	Planner	Executor	Perceptor
BreachSeek Alshehri et al. (2024)	Supervisor	Specialized agents	Recorder
cochise Happe and Cito (2025a)	Planner	Executor	Executor

Table 1: General Functions and Corresponding Agent Roles in AutoPT Frameworks

Framework	Reconnaissance	Retrieval	Agent orchestration	Reflect
ARACNE Nieponice et al. (2025)				Planner
AutoAttacker Xu et al. (2024)				Planner
AutoPentest Henke (2025)	Specialised Workers	Specialized Workers	Supervisor	Planner
CHECKMATE Wang et al. (2025a)				Planner
BreachSeek Alshehri et al. (2024)	Specialized Agents		Supervisor	Evaluator
cochise Happe and Cito (2025a)				Planner

Table 2: Dedicated Functions and Corresponding Agent Roles in AutoPT Frameworks

Figure 2: Taxonomy of planning strategies in AutoPT frameworks.

Panel of Table 3 — Table 3: Tool taxonomy for AutoPT agents. Reconnaissance Exploitation Both * Interactive

Model	Payload	Write ups	SSK	Search
VulnBot Kong et al. (2025b)	HackTricks	Hacking Articles
xOffense Luong et al. (2025)	HackTricks	Hacking Articles
hackingBuddyGPT Happe et al. (2023)	HackTricks
RapidPen Cowper (2024)	HackTricks
AI-Pentest-Benchmark Isozaki et al. (2025)	HackTricks
CYBER-ZERO Zhuo et al. (2026)		CTF write-ups

Table 4: PT LLM Agents and Frameworks Comparison

Figure 3: Illustration of various retrieval strategies. It highlights three primary mechanisms: dense retrieval based on vector embeddings, sparse retrieval focusing on exact entity keyword matching, and tool-based retrieval which delegates the retrieval autonomy to the LLM by selecting predefined testing methodologies from a library.

Name	Type	Data Source	Scale
A. CTF-Based Skill Assessment Benchmarks
InterCode-CTF Yang et al. (2023a)	CTF	picoCTF	100 CTF Tasks
NYU CTF Bench Shao et al. (2024b)	CTF	CSAW CTF	6 Scenarios (200 CTF Tasks)
CYBENCH Zhang et al. (2024a)	CTF	4 CTF Competitions	100 CTF Tasks
HackSynth Muzsai et al. (2024)	CTF	PicoCTF, OverTheWire	200 CTF Tasks
XBOW XBOW Engineering (2025)	CTF	–	104 CTF Tasks

Table 5: Comparison of Evaluation Benchmarks.

No.	Vulnerabilities	Description
Easy
005	IDOR, JWT, Default Credentials	IDOR, JWT misconfiguration, and default credentials enable privilege escalation
020	SSRF	SSRF enables unauthorized access to internal services
026	CVE, Path Traversal, LFI	Path traversal leads to local file inclusion and potential code execution
038	GraphQL, SQLi	SQL injection is present in a GraphQL endpoint
039	SQLi, Privilege Escalation	SQL injection enables authentication bypass and privilege escalation

Table 6: Detailed Characteristics of the Benchmark Challenges.

Panel of Table 7 — Table 7: Results of Medium and Hard challenges across frameworks.

Panel of Table 8 — Table 8: Results of Easy challenges across frameworks.

Panel of Table 9 — Table 9: Results of the KB ablation study on Medium and Hard challenges.

Panel of Table 10 — Table 10: Results of the KB ablation study on Easy challenges.

Panel of Table 11 — Table 11: Ablation study of backbone LLMs on Medium and Hard challenges.

Panel of Table 12 — Table 12: Ablation study of backbone LLMs on Easy challenges.

Figure 4: Tool usage distribution and call volume across frameworks by challenges difficulty.

Figure 5: Tool call distribution of major tools across backbone LLMs and frameworks.

Panel of Table 13 — Table 13: Comparison of Lite and Full CyberStrike configurations on Medium and Hard challenges.

Panel of Table 14 — Table 14: Comparison of Lite and Full CyberStrike configurations on Easy challenges.

Figure 6: Tool call composition and configuration effects in CyberStrike.

Figure 7: LLM calls and token consumption of each framework on successfully compromised challenges under Easy and Medium challenges.

Framework	Easy		Medium
Framework	Avg Time (s)	Success	Avg Time (s)	Success
CTFSOLVER	141	18	612	14
Cruiser	237	9	295	8
CyberStrike	286	14	903	9
baseline-kimi	321	16	1042	13
CHYing	390	11	551	6

Table 15: Average time and success count by frameworks.

Difficulty	Metric	Backbone LLM
Difficulty	Metric	Opus-4.6	Gemini-pro-3.1	DS-v3.2	DS-R-V3.2	GPT-5.2
CTFSOLVER
Easy	Success Count	18	18	18	18	18
	Calls	35	19.72	61.22	72.39	52.39
	Tokens	247437.56	153912.89	707196.22	498242.17	703328.17
	Time (s)	171	270	324	837	168

Table 16: Resource consumption of different backbone LLMs on CTFSOLVER and XBow-Comp across challenges difficulty.

Panel of Table 17 — Table 17: Capability stratification on challenge 022.

Panel of Table 18 — Table 18: Performance of single-agent frameworks with Opus-4.6.

Panel of Table 19 — Table 19: Capability stratification on challenge 026.

Framework	Challenges	Count
CHYing	014, 022, 028, 039, 041, 042, 060, 078, 093	009
H-Pentest	004, 022, 028, 029, 041, 093	006
SickHackShark	018, 022, 077, 093	004
Tinyctfer	026, 028, 029, 078, 088	005
CyberStrike	022, 028, 029	003
sub-agent	018	001

Table 20: Occurrences of flag hallucinations per framework.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Workflow of the CTFSOLVER framework.

Panel of Figure 9 — Figure 9: Framework card of CTFSOLVER.

Figure 10: Workflow of the LuaN1ao framework.

Panel of Figure 11 — Figure 11: Framework card of LuaN1ao.

Figure 12: Workflow of the Tinyctfer framework.

Panel of Figure 13 — Figure 13: Framework card of Tinyctfer.

Figure 14: Workflow of the XBow-Comp framework.

Panel of Figure 15 — Figure 15: Framework card of XBow-Competition.

Figure 16: Workflow of the Cruiser framework.

Panel of Figure 17 — Figure 17: Framework card of Cruiser.

Figure 18: Workflow of the CHYing framework.

Panel of Figure 19 — Figure 19: Framework card of CHYing.

Figure 20: Workflow of the SickHackShark framework.

Panel of Figure 21 — Figure 21: Framework card of SickHackShark.

Figure 22: Workflow of the newmapta framework.

Panel of Figure 23 — Figure 23: Framework card of newmapta.

Figure 24: Workflow of the sub-agent framework.

Panel of Figure 25 — Figure 25: Framework card of sub-agent.

Figure 26: Workflow of the CyberStrike framework.

Panel of Figure 27 — Figure 27: Framework card of CyberStrike.

Figure 28: Workflow of the H-Pentest framework.

Panel of Figure 29 — Figure 29: Framework card of H-Pentest.

Panel of Figure 30 — Figure 30: The prompt of baseline-kimi and baseline-cc

Automated Penetration Testing

Automated penetration testing uses software agents to find, verify, and assess exploitable weaknesses in systems.

See all 48 papers

APT-Agent: Automated Penetration Testing using Large Language Models
May 24, 2026William Guanting Li, Alsharif Abuadbba, Kristen Moore +1LLM Hallucination Mitigation Automated Penetration Testing
University of Queensland · University of Queensland Brisbane, Australia · CSIRO Data61 +2
ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense
Jun 4, 2026Anlan Zheng, Tiantian ZhuAI Agent Security Benchmarks Automated Penetration Testing
Zhejiang University of Technology Hangzhou, China
Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing
Jun 24, 2026Liwei Yu, Shuo Li, Ming Zhou +2LLM Agent Evaluation AI Agent Security Benchmarks
University of Science and Technology of China, Hefei, China · Nanjing University of Science and Technology, Nanjing, China · Research Institute, Runjian Co., Ltd, China +1

May 24, 2026J/K move · Enter open · S save

APT-Agent: Automated Penetration Testing using Large Language Models

William Guanting Li, Alsharif Abuadbba, Kristen Moore, Dan Dongseong Kim

University of Queensland · University of Queensland Brisbane, Australia · CSIRO Data61+2

LLM Hallucination Mitigation Automated Penetration Testing LLMs for Cybersecurity Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Penetration testing is essential to securing modern web infrastructures, yet traditional manual methods struggle to keep pace with their scale and complexity. Large Language Models (LLMs) offer new opportunities for automating these tasks, but existing approaches face two persistent challenges: hallucination of technical entities and insufficient long-term contextual memory. To address these issues, we present APT-Agent, a fully automated LLM-driven penetration testing framework that systematically orchestrates reconnaissance, exploitation, and exfiltration. APT-Agent introduces a hybrid rectification module to recover hallucinated commands and a command-specific memory architecture to preserve operational context across multi-step attack sequences. We evaluate our APT-Agent on Metasploitable 2 against seven vulnerable services spanning web, database, and network protocols. APT-Agent achieves an 84.29% end-to-end exploitation success rate, compared to 48.57% (Script Kiddie) and 18.57% (PentestGPT) under matched conditions. By reducing cognitive burden and minimizing reliance on human intervention, APT-Agent represents a step toward scalable, reliable, and cognitively efficient automation for penetration testing.

Abstract

Automated Penetration Testing

Explore similar work

APT-Agent: Automated Penetration Testing using Large Language Models

ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense

Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing