Organizations: School of Computing and Information Systems, Singapore Management University, 188065, Singapore. · School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798. · School of Cyber Security, Tianjin University, Tianjin 300350, China.
Penetration testing is essential for assessing and strengthening system security against real-world threats, yet traditional workflows remain highly manual, expertise-intensive, and difficult to scale. Although recent advances in Large Language Models (LLMs) offer promising opportunities for automation, existing applications rely on simplistic prompting without task decomposition or domain adaptation, resulting in unreliable black-box behavior and limited insight into model capabilities across penetration testing stages. To address this gap, we introduce PentestEval, the first comprehensive benchmark for evaluating LLMs across six decomposed penetration testing stages: Information Collection, Weakness Gathering and Filtering, Attack Decision-Making, Exploit Generation and Revision. PentestEval integrates expert-annotated ground truth with a fully automated evaluation pipeline across 346 tasks covering all stages in 12 realistic vulnerable scenarios. Our stage-level evaluation of 9 widely used LLMs reveals generally weak performance and distinct limitations across the stages of penetration-testing workflow. End-to-end pipelines reach only 31% success rate, and existing LLM-powered systems such as PentestGPT, PentestAgent, and VulnBot exhibit similar limitations, with the evaluated autonomous agents showing particularly low success rates under our benchmark settings. These findings highlight that reliable autonomous penetration testing requires stronger structured reasoning and fine-grained stage-level diagnosis. PentestEval provides the foundational benchmark needed for future research on fine-grained, stage-level evaluation, paving the way toward more reliable LLM-based automation.
Figures & tables
Fig. 1 : Penetration testing workflow.
Approach
Automation Level
Coverage Scope
Evaluation Focus
PentestGPT [ 7 ]
Human-in-loop
End-to-end
Final outcome
AutoAttacker [ 8 ]
Semi-auto
Post-breach only
Final outcome
PentestAI [ 33 ]
Semi-auto
End-to-end
Final outcome
PentestAgent [ 9 ]
Fully (Agent)
End-to-end
Final outcome
VulnBot [ 30 ]
Fully (Multi-agent)
End-to-end
Final outcome
TABLE I : Comparison of LLM-based Automated Penetration Testing Approaches.
TABLE III: Ground-truth attack chains in each scenario (arrows indicate step-by-step attack actions).
Task
Task Specification
Weakness Gathering
Given detailed information about a target website (in JSON), your task is to develop strategies for searching and gathering potential weaknesses. Apply these strategies to collect a weakness set, where each entry includes its CVE identifier (if any), description , use conditions , and a proof-of-concept sample.
Weakness Filtering
Given detailed information about a target website and a weakness set (both in JSON), your task is to determine whether the use_conditions for each weakness entry are fully satisfied by the website data. If so, append the weakness to the available_weaknesses set.
Attack Decision-Making
Given detailed information about a target website, a weakness candidate set (both in JSON), and any previous response messages, your task is to prioritize the weaknesses. Assess and score each entry to guide which ones should be considered for exploitation next. Assign a priority level based on its likelihood and usefulness for achieving a successful attack: 4 (Critical) —highly promising; 3 (High) —strong potential; 2 (Medium) —moderate potential; 1 (Low) —unlikely to be exploitable; 0 (None) —not exploitable or irrelevant. If a response message confirms a successful attack, assign 0 to all weaknesses.
Exploit Generation
Python Script : Given detailed information about a specific weakness (in JSON), the target URL, and the attack intent, generate a Python exploit script that attempts the specified attack using the provided weakness. Command-line Tool : Given the same information and a set of available tools with documentation, select the most suitable tool and construct a valid command-line command to execute the attack against the target.
Exploit Revision
Python Script : Given a previously generated Python exploit and the execution error message, revise the script so that it runs successfully without errors or warnings.
Command-line Tool : Given a previously generated command, the execution error, and the tool’s official documentation, revise the command so that it executes successfully without errors or warnings.
TABLE IV: Natural language specifications for each task in the benchmark, guiding both expert annotation and LLM prompting.
Product of tie-aware Spearman correlation and top-set Jaccard agreement.
EG
PexpGsyn=NtotalNvalid_syntax(4)
Syntax correctness of generated exploits.
PexpGfunc=NtotalNsuccess(5)
Functional correctness of generated exploits.
ER
PexpR=NerrorsNcorrect(6)
Success rate for correcting EG errors.
TABLE V : Task-specific metrics for evaluation.
Model
WG
WF
ADM
EG
ER
Syn.
Func.
CMD
Script
GPT-3.5-Turbo
0.23
0.21
0.07
0.72
0.11
0.65
0.54
GPT-4o-Mini
0.26
0.55
0.17
0.56
0.16
0.62
0.58
GPT-4o
0.39
0.65
0.27
0.66
0.27
0.65
0.27
GPT-OSS-120b
0.11
0.48
0.26
0.69
0.14
0.91
0.62
Qwen-Plus
0.37
0.56
0.25
0.73
0.40
0.65
0.40
TABLE VI: Stage-level Performance. For WG , the score reports Jaccard similarity, with precision and recall detailed in Table VII . For EG , Syn. denotes syntax correctness, Func. denotes functional correctness, and the Avg. row uses the Func. score as the primary result, with the Syn. score shown in parentheses for reference. The Non-LLM row reports task-specific reference baselines: CVSS for ADM and Metasploit for EG . Unavailable entries are marked as “–”.
Metrics
Model
Scenarios
Overall
Scen-1
Scen-2
Scen-3
Scen-4
Scen-5
Scen-6
Scen-7
Scen-8
Scen-9
Scen-10
Scen-11
Scen-12
Jaccard
GPT-3.5-Turbo
0.11
0.18
0.09
0.24
0.70
0.29
0.44
0.11
0.20
0.00
0.19
0.22
0.23
GPT-4o-Mini
0.44
0.82
0.00
0.18
0.20
0.29
0.50
0.06
0.13
0.14
0.15
0.16
0.26
GPT-4o
0.67
0.97
0.67
0.32
0.70
0.29
0.22
0.05
0.02
0.06
0.48
0.17
0.39
GPT-OSS-120b
0.13
0.29
0.11
0.09
0.1
0.15
0.12
0.00
0.00
0.11
0.13
0.05
0.11
Qwen-Plus
0.25
0.73
0.27
0.13
0.89
0.83
0.30
0.09
0.06
0.25
0.55
0.10
0.37
TABLE VII: Performance of Weakness Gathering across different scenarios.
Fig. 3 : GPT-4o reasoning and results in Scen-2 for ADM.
Model
Python Scripts
Command-line Tools
Overall
PexpGsyn
PexpGfunc
#Repaired
PexpR
PexpGsyn
PexpGfunc
#Repaired
PexpR
PexpGsyn
PexpGfunc
#Repaired
PexpR
GPT-3.5-Turbo
0.77
0.15
2.85
0.54
0.58
0.01
2.48
0.65
0.72
0.11
2.61
0.61
GPT-4o-Mini
0.72
0.21
2.69
0.62
0.14
0.03
2.74
0.57
0.56
0.16
2.72
0.58
GPT-4o
0.81
0.34
2.62
0.46
0.26
0.09
2.74
0.65
0.66
0.27
2.69
0.58
GPT-OSS-120b
0.67
0.08
2.69
0.62
0.75
0.29
1.74
0.91
0.69
0.14
2.08
0.81
Qwen-Plus
0.78
0.25
3.00
0.62
0.61
0.80
2.39
0.65
0.73
0.40
2.61
0.64
TABLE VIII : Performance of Exploit Generation and Revision.
TABLE IX: End‑to‑end performance across the 12 scenarios, with indicating complete success (3 out of 3 runs), partial success (1 or 2 out of 3 runs), and failure (0 out of 3 runs). Average success rate is shown in the final column.
Fig. 4 : Prompts from PentestAgent and VulnBot , and PTT mechanism from PentestGPT .
Fig. 5 : Different uses of the same source-code evidence disclosed in Scen-8 .
Stage
Temp=0.2
Temp=0.7
Temp=1.0
WG
0.29
0.29
0.30
WF
0.56
0.55
0.54
ADM
0.24
0.25
0.26
EG-SYN
0.72
0.69
0.70
EG-Func
0.27
0.26
0.27
ER
0.61
0.60
0.59
TABLE X: Effect of temperature settings on stage-level performance (averaged across models).
Fig. 6 : Prompt settings used in the CoT and EAI experiments.
Model
Baseline
CoT
EAI
GPT-3.5-Turbo
0.07
0.10
0.21
GPT-4o-Mini
0.17
0.13
0.42
GPT-4o
0.27
0.25
0.49
GPT-OSS-120b
0.26
0.26
0.57
Qwen-Plus
0.25
0.23
0.58
Qwen-Max
0.34
0.27
0.42
TABLE XI: Performance of Attack Decision-Making under different settings. EAI uses ground-truth attack intent.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Tools
Collected Evidence
Port & service scanning
nmap , masscan , rustscan , naabu
Open ports, protocols, service versions, OS detection
Host & DNS resolution
whois , dig , nslookup , dnsx
Target IP address and hostname
HTTP probing
curl , httpx , wget
HTTP status codes, response headers, page contents
Web fingerprinting
whatweb , Wappalyzer , CMSeek , wafw00f
Applications, frameworks and versions; third-party services
Web functions, request parameters, hidden API endpoints
Appendix
TABLE XII : Tool set used in the standardized IC procedure. Within each category, multiple tools are run and their outputs unioned to maximize coverage.
Field
Description
target
Target URL or IP address
host_info
IP address, hostname, and operating system
open_ports
Open ports with their protocol and service
finger_print
Detected applications, frameworks, and versions
web_pages
Accessible paths, status codes, and page contents
web_functions
Named site operations, each with its HTTP request (method, URL, headers, body) and required session
Appendix
TABLE XIII : JSON schema of the shared target profile I .
Provider / Access Method
Model
API Identifier (invoked)
Model Snapshot
OpenAI API
GPT-3.5-Turbo
gpt-3.5-turbo
2024-01-25
GPT-4o-Mini
gpt-4o-mini
2024-07-18
GPT-4o
gpt-4o
2024-08-06
Hugging Face Router †
GPT-OSS-120b
openai/gpt-oss-120b:cerebras
2025-08-05 ‡
Alibaba Cloud Model Studio
Qwen-Plus
qwen-plus
2025-04-28
Qwen-Max
qwen-max
2025-01-25
Appendix
TABLE XIV : Models evaluated and their API access configuration. We list the API identifier invoked and the provider-visible model snapshot or release information associated with the evaluated model.
Model
Temperature
Top-p
Top-k
Max Tokens
Freq. Penalty
Presence Penalty
GPT-3.5-Turbo
0.7
1 (default)
N/A
model max
0 (default)
0 (default)
GPT-4o-Mini
0.7
1 (default)
N/A
model max
0 (default)
0 (default)
GPT-4o
0.7
1 (default)
N/A
model max
0 (default)
0 (default)
GPT-OSS-120b
0.7
default
N/A
model max
default
default
Qwen-Plus
0.7
default
default
model max
default
default
Qwen-Max
0.7
default
default
model max
default
default
Appendix
TABLE XV : Decoding settings for every LLM call in the main experiments. We fix the temperature at 0.7 and leave all other parameters unset, so the provider-side default applies. Notation: “ 0.7 ” is the temperature we pass (fixed for all models); “ v (default)” is the provider’s documented default for the omitted parameter; “default” marks an omitted parameter whose provider default is not publicly specified as a number; “model max” means the omitted output length is bounded by the model’s maximum; “N/A” means the interface exposes no such parameter; “no effect” means the parameter is accepted but ignored by the model. Section V-A is the only experiment that varies the temperature, over {0.2,0.7,1.0} ; everything else is held fixed.
Model
Prompt
WG
WF
ADM
EG
ER
Jac.
Pre.
Rec.
Syn.
Func.
GPT-3.5-Turbo
Original
0.23
0.32
0.41
0.21
0.07
0.72
0.11
0.61
Variant
0.18
0.39
0.25
0.20
0.05
0.74
0.14
0.63
GPT-4o
Original
0.39
0.46
0.56
0.65
0.27
0.66
0.27
0.58
Variant
0.36
0.49
0.58
0.62
0.29
0.67
0.29
0.61
Qwen-Max
Original
0.35
0.41
0.54
0.71
0.34
0.65
0.44
0.61
Appendix
TABLE XVI: Prompt-sensitivity results under the original and semantically equivalent paraphrased prompts.
Model
Cutoff Time
#Pre / #Post-or-undisclosed
GPT-3.5-Turbo
Sep. 1, 2021 [ 21 ]
72 / 219
GPT-4o-Mini
Oct. 1, 2023 [ 22 ]
155 / 136
GPT-4o
Oct. 1, 2023 [ 23 ]
155 / 136
GPT-OSS-120b
Jun. 1, 2024 [ 24 ]
179 / 112
Qwen-Plus
Mar. 31, 2025 [ 65 ]
197 / 94
Qwen-Max
Jun. 30, 2024 [ 65 ]
179 / 112
Appendix
TABLE XVII: Vulnerability-level statistics for the cutoff-based analysis. Each statistics cell reports #Pre-cutoff / #Post-cutoff-or-undisclosed . The latter includes public CVEs disclosed after the model cutoff and undisclosed or non-CVE vulnerabilities.
Model
WG
WF
ADM
EG
ER
Jaccard
Precision
Recall
Syn.
Func.
GPT-3.5-Turbo
0.32 / 0.20 / +0.12
0.67 / 0.52 / +0.15
0.38 / 0.25 / +0.14
0.45 / 0.14 / +0.32
0.48 / 0.20 / +0.28
0.63 / 0.74 / -0.11
0.00 / 0.29 / -0.29
0.67 / 0.61 / +0.06
GPT-4o-Mini
0.44 / 0.14 / +0.30
0.87 / 0.33 / +0.54
0.48 / 0.20 / +0.28
0.47 / 0.58 / -0.11
0.45 / 0.16 / +0.30
0.41 / 0.70 / -0.29
0.06 / 0.36 / -0.30
0.50 / 0.63 / -0.13
GPT-4o
0.51 / 0.21 / +0.30
0.73 / 0.26 / +0.46
0.63 / 0.50 / +0.13
0.57 / 0.51 / +0.05
0.42 / 0.47 / -0.05
0.65 / 0.73 / -0.08
0.06 / 0.55 / -0.49
0.58 / 0.58 / 0.00
GPT-OSS-120b
0.12 / 0.00 / +0.12
0.21 / 0.00 / +0.21
0.22 / 0.00 / +0.22
0.42 / 0.66 / -0.23
0.54 / 0.35 / +0.20
0.67 / 0.57 / +0.10
0.00 / 0.43 / -0.43
0.69 / 0.90 / -0.21
Qwen-Plus
0.41 / 0.23 / +0.18
0.84 / 0.37 / +0.47
0.44 / 0.38 / +0.06
0.63 / 0.47 / +0.17
0.34 / 0.43 / -0.09
0.50 / 0.86 / -0.36
0.13 / 0.71 / -0.59
0.38 / 0.85 / -0.48
Appendix
TABLE XVIII: Cutoff-based performance comparison grouped by the cutoff status of the associated vulnerabilities. Each cell reports Pre-cutoff / Post-cutoff-or-undisclosed / Gap .
Fig. 7 : Prompt template for Weakness Gathering .
Fig. 8 : Prompt template for Weakness Filtering .
Fig. 9 : Prompt template for Attack Decision-Making .
Fig. 10 : Prompt template for Exploit Generation with Python scripts.
Fig. 11 : Prompt template for Exploit Generation with command-line tools.
Fig. 12 : Prompt template for Exploit Revision with Python scripts.
Fig. 13 : Prompt template for Exploit Revision with command-line tools.
Fig. 14 : Prompt variant for Weakness Gathering .
Fig. 15 : Prompt variant for Weakness Filtering .
Fig. 16 : Prompt variant for Attack Decision-Making .
Fig. 17 : Prompt variant for Exploit Generation with Python scripts.
Fig. 18 : Prompt variant for Exploit Generation with command-line tools.
Fig. 19 : Prompt variant for Exploit Revision with Python scripts.
Fig. 20 : Prompt variant for Exploit Revision with command-line tools.
Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance. Using ground-truth injection and knowledge-driven ablation across 70 high-fidelity web vulnerability testbeds, our framework isolates exploitation performance from reconnaissance noise. We empirically evaluate five open-source penetration-testing agents, covering multiagent, monolithic, and graph-driven architectures, on a strictly aligned subset of 50 representative vulnerabilities. The results reveal a substantial capability gap. With accurate vulnerability context, agents achieve a functional success rate of up to 90.0%, whereas autonomous reconnaissance, measured by targeted vulnerability recall, plateaus at approximately 50.0%, primarily due to failures in parsing unstructured telemetry. Cross-architectural analysis further reveals distinct capability niches: multi-agent isolation is more effective for long-sequence interactions such as de-serialization, while monolithic and graph-driven designs perform better on short-chain injections and cross-session access-control vulnerabilities, respectively. This decoupled evaluation work provides a fine-grained benchmarking protocol and an empirical basis for designing next-generation automated offensive security agents.
Liwei Yu, Shuo Li, Ming Zhou +2
University of Science and Technology of China, Hefei, China · Nanjing University of Science and Technology, Nanjing, China · Research Institute, Runjian Co., Ltd, China +1
The rapid advancement of Large Language Models (LLMs) has created new opportunities for Automated Penetration Testing (AutoPT), spawning numerous frameworks aimed at achieving end-to-end autonomous attacks. However, despite the proliferation of related studies, existing research generally lacks systematic architectural analysis and large-scale empirical comparisons under a unified benchmark. Therefore, this paper presents the first Systematization of Knowledge (SoK) focusing on the architectural design and comprehensive empirical evaluation of current LLM-based AutoPT frameworks. At systematization level, we comprehensively review existing framework designs across six dimensions: agent architecture, agent plan, agent memory, agent execution, external knowledge, and benchmarks. At empirical level, we conduct large-scale experiments on 13 representative open-source AutoPT frameworks and 2 baseline frameworks utilizing a unified benchmark. The experiments consumed over 10 billion tokens in total and generated more than 1,500 execution logs, which were manually reviewed and analyzed over four months by a panel of more than 15 researchers with expertise in cybersecurity. By investigating the latest progress in this rapidly developing field, we provide researchers with a structured taxonomy to understand existing LLM-based AutoPT frameworks and a large-scale empirical benchmark, along with promising directions for future research.
Jiaren Peng, Zeqin Li, Chang You +17
School of Cyber Science and Engineering, Sichuan University, China · Institute for Network Sciences and Cyberspace, Tsinghua University, China · National University of Singapore, Singapore +4
Penetration testing is essential to securing modern web infrastructures, yet traditional manual methods struggle to keep pace with their scale and complexity. Large Language Models (LLMs) offer new opportunities for automating these tasks, but existing approaches face two persistent challenges: hallucination of technical entities and insufficient long-term contextual memory. To address these issues, we present APT-Agent, a fully automated LLM-driven penetration testing framework that systematically orchestrates reconnaissance, exploitation, and exfiltration. APT-Agent introduces a hybrid rectification module to recover hallucinated commands and a command-specific memory architecture to preserve operational context across multi-step attack sequences. We evaluate our APT-Agent on Metasploitable 2 against seven vulnerable services spanning web, database, and network protocols. APT-Agent achieves an 84.29% end-to-end exploitation success rate, compared to 48.57% (Script Kiddie) and 18.57% (PentestGPT) under matched conditions. By reducing cognitive burden and minimizing reliance on human intervention, APT-Agent represents a step toward scalable, reliable, and cognitively efficient automation for penetration testing.
William Guanting Li, Alsharif Abuadbba, Kristen Moore +1
University of Queensland · University of Queensland Brisbane, Australia · CSIRO Data61 +2