cs.SEDec 16, 2025

PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design

Authors: Ruozhao Yang, Mingfei Cheng, Gelei Deng, Tianwei Zhang, Junjie Wang, Xiaofei Xie

Organizations: School of Computing and Information Systems, Singapore Management University, 188065, Singapore. · School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798. · School of Cyber Security, Tianjin University, Tianjin 300350, China.

Abstract

Penetration testing is essential for assessing and strengthening system security against real-world threats, yet traditional workflows remain highly manual, expertise-intensive, and difficult to scale. Although recent advances in Large Language Models (LLMs) offer promising opportunities for automation, existing applications rely on simplistic prompting without task decomposition or domain adaptation, resulting in unreliable black-box behavior and limited insight into model capabilities across penetration testing stages. To address this gap, we introduce PentestEval, the first comprehensive benchmark for evaluating LLMs across six decomposed penetration testing stages: Information Collection, Weakness Gathering and Filtering, Attack Decision-Making, Exploit Generation and Revision. PentestEval integrates expert-annotated ground truth with a fully automated evaluation pipeline across 346 tasks covering all stages in 12 realistic vulnerable scenarios. Our stage-level evaluation of 9 widely used LLMs reveals generally weak performance and distinct limitations across the stages of penetration-testing workflow. End-to-end pipelines reach only 31% success rate, and existing LLM-powered systems such as PentestGPT, PentestAgent, and VulnBot exhibit similar limitations, with the evaluated autonomous agents showing particularly low success rates under our benchmark settings. These findings highlight that reliable autonomous penetration testing requires stronger structured reasoning and fine-grained stage-level diagnosis. PentestEval provides the foundational benchmark needed for future research on fine-grained, stage-level evaluation, paving the way toward more reliable LLM-based automation.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing

    Jun 24, 2026Liwei Yu, Shuo Li, Ming Zhou +2LLM Agent EvaluationAI Agent Security Benchmarks

  2. Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing

    Apr 7, 2026Jiaren Peng, Zeqin Li, Chang You +17Automated Penetration TestingLLMs for Cybersecurity

  3. APT-Agent: Automated Penetration Testing using Large Language Models

    May 24, 2026William Guanting Li, Alsharif Abuadbba, Kristen Moore +1LLM Hallucination MitigationAutomated Penetration Testing