Automated Penetration Testing

Follow this topic Ask about this topic

Automated penetration testing studies how software agents can discover vulnerabilities, plan attack paths, and verify exploits with limited human direction. Approaches range from LLM agents that use security tools and persistent memory to reinforcement-learning policies, multi-agent workflows, and symbolic attack graphs. Researchers evaluate them on CTFs, vulnerable web applications, and simulated or live cyber ranges, measuring validated findings, task completion, cost, and scope adherence. A central challenge is making agents reliable across unfamiliar, multi-step targets while ensuring their findings are verified and their actions remain within authorized boundaries.

Latest papers

48 papers4 papers in the last four weeks, down 56% on the four weeks before. 0.0% of all new papers.

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Oct 13, 2026Bangya Liu, Jianhong Tu, Yueyang Pan +19AI Agent Security Benchmarks Cybersecurity
2077AI Open Source Foundation · University of Wisconsin–Madison · UC Santa Cruz +8
Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications
Oct 13, 2026Yang Luo, Tiantian Ji, Zifeng Kang +4Software Security Prompt Injection Attacks
Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications · Beijing University of Posts and Telecommunications · Zhongguancun Laboratory +1
ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
Oct 13, 2026Fasheng Miao, Zhaoyu Hu, Xiang Li +5AI Agent Security Benchmarks LLM Agent Security
Tsinghua University · Alibaba Group · Nankai University +1
AgentTrap: Stateful Feedback Deception against Autonomous Penetration Testing Agents
Oct 2, 2026Yuelin Wang, Jiongchi Yu, Yanbang SunCybersecurity AI Agent Security
Tianjin University Tianjin, China · Nanyang Technological University Singapore
AgentXploit: Autonomous Repository-to-Runtime Red-Teaming for AI Agents
Sep 25, 2026Weida Liang, Shi Qiu, Zhun Wang +6AI Agent Auditing Automated Red Teaming
National University of Singapore · University of North Carolina at Chapel Hill · University of California, Berkeley +2
Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents
Sep 24, 2026Joas Antonio dos Santos BarbosaProbability Calibration Cybersecurity
Independent Researcher — AI and Offensive Security S˜ao Paulo, SP, Brazil
ScopeBench: Do Agents Preserve Engagement Boundaries Under Goal Pressure?
Sep 23, 2026Shane Caldwell, Max Harley, Ads Dawson +3AI Agent Evaluation AI Agent Security Benchmarks
dreadnode, USA
Evaluating Coding Agents on Kernel Exploit Generation
Sep 22, 2026Junyoung Jang, Gwanhyun Lee, Hwiwon Lee +4AI Coding Agents Software Security
Independent Researcher · University of Illinois Urbana-Champaign · Ministry of National Defense, Republic of Korea
MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
Sep 21, 2026Andy K. Zhang, Ava Huang, Joey Ji +21Software Vulnerability Detection AI Agent Security Benchmarks
Stanford University · UC Berkeley
From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation
Sep 19, 2026Joas Antonio dos Santos BarbosaAutomated Penetration Testing Runtime Enforcement for AI Agents
Independent Researcher — AI and Offensive Security S˜ao Paulo, SP, Brazil
PentestChain: A Cost-Aware, MCP-Orchestrated Framework for Automated Penetration Testing with Free-Tier LLMs
Sep 16, 2026Rushabh Vipulkumar Patel, Dipo Dunsin, Mohammed Almaiah +1Automated Penetration Testing LLMs for Cybersecurity
Computer Security Laboratory, University of Wales Trinity Saint David, London, United Kingdom · King Abdullah IT School, The University of Jordan, Amman, Jordan · School of Computer Science and Mathematics, Keele University, Newcastle-under-Lyme, United Kingdom
Automating Attack Graph Construction for Agentic Pentesting. Towards Neuro-Symbolic Vulnerability Hunting
Sep 14, 2026Oliver Stevanovic, Jasmin WachterAutomated Penetration Testing Neuro-Symbolic AI
University of Udine via Palladio 833100 Udine, Italy · University of Klagenfurt Universitätsstraße 65-67, 9020 Klagenfurt, Austria
Staying on the Attack Path: Structured State for Long-Horizon Automated Penetration Testing
Sep 7, 2026Weizhe Wang, Yitong Zhang, Yao Zhang +4LLM Agent Memory LLM Agent Security
Tianjin University, Tianjin, 300072, China · Changchun University of Science and Technology, Changchun, 130022, China · Jilin Province Key Laboratory of Network and Information Security, Changchun, 130022, China +1
Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing
Aug 16, 2026Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu +3LLM Agent Security Automated Penetration Testing
Department of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh · Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh +3
SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary system
Aug 15, 2026Yuhan Meng, Shaofei Li, Jionghao Huang +8Cybersecurity Autonomous Cyber Defense
School of Computer Science, Peking University · Southeast University · School of Computing, National University of Singapore
RangeFactory: Scalable Construction of Multi-Hop Cyber Ranges
Aug 10, 2026Hanlin Jiang, Puyi Wang, Jiandong Jin +10Cybersecurity Automated Penetration Testing
Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education · Huawei Technologies Co., Ltd. · Southeast University
AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents
Jul 29, 2026Ruoyu Wang, Heng Zhao, Renjie Wu +4Autonomous Cyber Defense Cybersecurity
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · 3Hainan University · 4Anhui University +1
StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents
Jul 28, 2026Ads Dawson, Adrian WoodAI Agent Security LLM Agent Security
Security Researcher, USA
Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents
Jul 22, 2026Or Zion Eliav, Eyal Lenga, Shir Bernstien +1AI Agent Security AI Agent Security Benchmarks
Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer-Sheva, Israel
Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing
Jul 13, 2026Ananda Dhakal, Krish Neupane, Aarjan ChaudharyAI Agent Evaluation AI Agent Security Benchmarks
Kroda Labs
Mako: A Self-Evolving Agentic Operating System (SE-AOS) for Autonomous Web Exploitation
Jul 13, 2026Praneeth Narisetty, Shiva Nagendra Babu KoreAutomated Penetration Testing Automated Red Teaming
LaunchSafe
VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents
Jul 10, 2026Katherine Swinea, Kshitiz Aryal, Lopamudra Praharaj +1Cybersecurity LLM Agent Security
Department of Computer Science, Tennessee Tech University, TN, USA. · School of Interdisciplinary Informatics, University of Nebraska Omaha, NE, USA. · Dept. of Mathematics & Computer Science, University of North Carolina at Pembroke, NC, USA.
Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing
Jun 24, 2026Liwei Yu, Shuo Li, Ming Zhou +2LLM Agent Evaluation AI Agent Security Benchmarks
University of Science and Technology of China, Hefei, China · Nanjing University of Science and Technology, Nanjing, China · Research Institute, Runjian Co., Ltd, China +1
Red-Teaming the Agentic Red-Team
Jun 23, 2026Dario Pasquini, Michal Bazyli, Taras Fedynyshyn +1Software Security AI Agent Security
Cracken · Information Protection Department, Lviv Polytechnic National University, Lviv, 79013, Ukraine
AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
Jun 12, 2026Fengyu Liu, Jiarun Dai, Yihe Fan +11Cybersecurity AI Agent Security Benchmarks
Fudan University
The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems
Jun 11, 2026Jiaqi Luo, Jiarun Dai, Zhile Chen +8LLM Evaluation AI Agent Security Benchmarks
Fudan University · Shanghai Artificial Intelligence Laboratory · Concordia AI +1
ZERO-APT: A Closed-Loop Adversarial Framework for LLM-Driven Automated Penetration Testing under Intelligent Defense
Jun 4, 2026Anlan Zheng, Tiantian ZhuAI Agent Security Benchmarks Automated Penetration Testing
Zhejiang University of Technology Hangzhou, China
FORGE: Multi-Agent Graduated Exploitation and Detection Engineering
Jun 2, 2026Farooq ShaikhAutomated Exploit Generation Cybersecurity
Dynatrace, Vienna, Austria
How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency
May 28, 2026Galip Tolga ErdemAutomated Penetration Testing LLMs for Cybersecurity
Independent Researcher
CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Fly
May 25, 2026Yihe Fan, Changyi Li, Lichen Xu +4LLM Agent Self-Improvement Automated Penetration Testing
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Pudong Research Institute of Cryptology

Oct 13, 2026J/K move · Enter open · S save

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

Bangya Liu, Jianhong Tu, Yueyang Pan, Hanhong Zhao+18

2077AI Open Source Foundation · University of Wisconsin–Madison · UC Santa Cruz+8

AI Agent Security Benchmarks Cybersecurity LLMs for Cybersecurity Autonomous Cyber Defense Automated Penetration Testing Adversarial Scenario Generation

Open paper AskPDF arXiv

AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static benchmarks quickly saturate and miss an agent's ability to attack and defend simultaneously under live pressure. To address this gap, we present CyberArena, a dynamic, symmetric adversarial benchmark built around multi-round attack-and-defense combat. In isolated sandboxes, two agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an agent's speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain the benchmark's longevity, an automated agentic pipeline continuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce a dynamical model that maps match dynamics into unified, multidimensional scores across offensive, defensive, and domain-specific capabilities. Evaluating 12 frontier LLM agents across 40 challenges in 466 matches, CyberArena separates model capabilities where single-agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a benchmark, CyberArena serves as an adaptable, scalable experimental platform for future research on cybersecurity agents.