AI Agent Security Benchmarks

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 91% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 200

All topics

Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
May 12, 2026Chang Jin, An Wang, Zeming Wei +7LLM Agent Security AI Agent Security Benchmarks
1Shanghai AI Laboratory · 2Peking University · 3East China Normal University
IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection
May 12, 2026Chia-Pei, Chen, Kentaroh Toyoda +2AI Agent Security AI Agent Security Benchmarks
Janet · Vulcan Research, AIFT
Cochise: A Reference Harness for Autonomous Penetration Testing
May 12, 2026Andreas Happe, Jürgen CitoLLM Agent Orchestration LLM Agent Evaluation
TU Wien · Vienna, Austria
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
May 12, 2026Dongjun Lee, Ga-eun Bae, Insu YunBenchmark Contamination LLM Agent Evaluation
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
May 11, 2026Zhun Wang, Nico Schiller, Hongwei Li +13AI Agent Security AI Agent Security Benchmarks
1UC Berkeley · 2Max Planck Institute for Security and Privacy · 3UC Santa Barbara +4
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
May 11, 2026Pedro Conde, Henrique Branquinho, Valerio Mazzone +3Software Vulnerability Detection AI Agent Evaluation
Ethiack Coimbra, Portugal · Ethiack Porto, Portugal · University of Notre Dame Notre Dame, Indiana, USA
LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
May 11, 2026Chiyu Zhang, Huiqin Yang, Bendong Jiang +8AI Agent Security Benchmarks LLM Jailbreak Attacks
Nanjing University of Aeronautics and Astronautics · Collaborative lnnovation Center of Novel Software Technology and Industrialization · Zhejiang University
CrackMeBench: Binary Reverse Engineering for Agents
May 11, 2026Isaac David, Arthur GervaisSoftware Reverse Engineering LLM Agent Evaluation
University College London
Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
May 11, 2026Hongwei Yao, Yiming Liu, Yiling He +1AI Agent Security LLM Agent Security
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · University College London, London, United Kingdom +1
MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
May 10, 2026Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews +1Software Engineering Agents AI Agent Security Benchmarks
Independent · Redwood Research
FORTIS: Benchmarking Over-Privilege in Agent Skills
May 9, 2026Shawn Li, Chenxiao Yu, Han Wang +8LLM Agent Security AI Agent Security Benchmarks
University of Southern California · University of Illinois at Urbana-Champaign · 3Adobe Research +3
Agent Collectives Should Not Detect Their Own Imposters: A Chess Case Study
May 9, 2026Alexandre Le Mercier, Chris Develder, Thomas DemeesterAI Agent Security AI Agent Security Benchmarks
IDLab–T2K, Ghent University–imec
CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
May 8, 2026Taein Lim, Seongyong Ju, Munhyeok Kim +2AI Agent Security Benchmarks LLMs for Cybersecurity
Chung-Ang University · Myongji University
Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems
May 6, 2026Krti TallamAgent Failure Analysis Evidence-Grounded Reasoning
KamiwazaAI
DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
May 6, 2026Zhaorun Chen, Xun Liu, Haibo Tong +14AI Agent Evaluation AI Agent Security
Virtue AI · University of Chicago · University of Illinois, Urbana-Champaign +4
AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use
May 6, 2026Chenglin YangAI Agent Security AI Agent Security Benchmarks
Independent Researcher
MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
May 5, 2026Jonathan Steinberg, Oren GalLanguage Model Safety Evaluation AI Coding Agents
Swarms & AI Lab (SAIL), University of Haifa
Trojan Hippo Bench: A Dynamic Benchmark for Persistent Memory Attacks and Defenses in LLM Agents
May 3, 2026Debeshee Das, Julien Piet, Darya Kaviani +3LLM Agent Security AI Agent Security Benchmarks
ETH Zürich · UC Berkeley · Snyk
Architectural Obsolescence of Unhardened Agentic-AI Runtimes
May 3, 2026Alfredo MetereAI Agent Security AI Agent Security Benchmarks
Metere Consulting, LLC.
MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents
Apr 30, 2026Haonan Li, Tianjun Sun, Yongqing Wang +1MCP Security Privacy Leakage in Language Models
Key Laboratory of Intraplate Volcanoes and Earthquakes (China University of Geosciences, Beijing), Ministry of Education, Beijing 100083, China · School of Geophysics and Information Technology, China University of Geosciences, Beijing 100083, China
Autonomous LLM Agents & CTFs: A Second Look
Apr 29, 2026Youness Bouchari, Matteo Boffa, Marco Mellia +3LLM Agent Orchestration AI Agent Security Benchmarks
Politecnico di Torino · Universit`a di Torino · Huawei Technologies France
Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skills
Apr 28, 2026Lijia Lv, Xuehai Tang, Jie Wen +2AI Agent Auditing AI Agent Security
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systems
Apr 27, 2026Pablo Mateo-Torrejón, Alfonso Sánchez-MaciánMulti-Agent LLM Systems AI Agent Security Benchmarks
University Carlos III of Madrid · Leganés, Spain
A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework
Apr 25, 2026Kexin ChuAI Agent Security LLM Agent Security
School of Computing, University of Connecticut
Beyond Single-Agent Alignment: Preventing Context-Fragmented Violations in Multi-Agent Systems
Apr 24, 2026Jie Wu, Ming GongCybersecurity AI Agent Security Benchmarks
Atlassian
CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents
Apr 23, 2026Wenjie Fu, Xiaoting Qin, Jue Zhang +5Data Leakage Contextual Integrity
Huazhong University of Science and Technology, China · Microsoft
Cross-Session Threats in AI Agents: Benchmark, Evaluation, and Algorithms
Apr 22, 2026Ari AzarafroozCybersecurity AI Agent Security
Intrinsec AI
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Apr 21, 2026Alankrit Chona, Igor Kozlov, Ambuj KumarAI Agent Security Benchmarks LLMs for Cybersecurity
Simbian AI
Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges
Apr 21, 2026Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3LLM Agent Evaluation AI Agent Security Benchmarks
Delft University of Technology Delft, The Netherlands

May 12, 2026, cs.AIJ/K move · Enter open · S save

Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung+2

UC Berkeley

Computer-Use Agent Benchmarks Reward Hacking Benchmark Auditing AI Agent Auditing AI Agent Security Benchmarks AI Agent Benchmarks+3

Open paper AskPDF arXiv

Agent benchmarks have become the de facto measure of frontier AI competence, guiding model selection, investment, and deployment. However, reward hacking, where agents maximize a score without performing the intended task, emerges spontaneously in frontier models without overfitting. We argue that benchmarks must be secure by design. From past incidents of reward hacks, we derive a taxonomy of eight recurring flaw patterns and compile them into the Agent-Eval Checklist for benchmark designers. We condense the insights into BenchJack, an automated red-teaming system that drives coding agents to audit benchmarks and identify possible reward-hacking exploits in a clairvoyant manner. Moreover, we extend BenchJack to an iterative generative-adversarial pipeline that discovers new flaws and patches them iteratively to improve benchmark robustness. We apply BenchJack to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations. BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task, surfacing 219 distinct flaws across the eight classes. Moreover, BenchJack's extended pipeline reduces the hackable-task ratio from near 100% to under 10% on four benchmarks without fatal design flaws, fully patching WebArena and OSWorld within three iterations. Our results show that evaluation pipelines have not internalized an adversarial mindset, and that proactive auditing could help close the security gap for the fast-paced benchmarking space.

Momentum

Related topics

Latest papers 200