Automated Agent Evaluation

Latest papers

41 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent Evaluation Computer-Use Agent Benchmarks
The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14
Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
Oct 10, 2026Tianyu Ding, Yaocheng Tong, Muzhi Yang +3Benchmark Reproducibility Automated Agent Evaluation
Independent Researchers
DelegationBench: Measuring When AI Agents Should Ask Before Acting
Oct 4, 2026Shiva PochampallyLLM Agent Evaluation AI Agent Benchmarks
Towards Automatically Pruning Logging Code with Coding Agents: How Far Are We?
Oct 3, 2026He Yang Yuan, Haonan Zhang, Xin Wang +4Automated Agent Evaluation Coding Agents
York University Toronto, Canada · University of Waterloo Waterloo, Canada · HKUST (Guangzhou) Guangzhou, China +2
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
Oct 1, 2026Luis Wiedmann, Leander Girrbach, Cordelia Schmid +1AI Agent Reliability AI Agent Evaluation
Technical University of Munich, MCML · Helmholtz Munich · Inria, École normale supérieure, CNRS, PSL Research University
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Sep 24, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Knowledge-Intensive QA LLM Agent Evaluation
Eon
Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error
Sep 13, 2026Hongliu CaoAI Agent Evaluation Automated Agent Evaluation
Amadeus SAS
Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks
Sep 10, 2026Mia Lassiter, Brinnae BentAI Agent Evaluation AI Agent Benchmarks
Duke University
Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
Sep 8, 2026Mohsen ArjmandiCoding Agents LLM Agent Evaluation
evolutionID GmbH, Germany
BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
Aug 27, 2026Mesut TorukLLM Agent Skill Evaluation LLM Agent Evaluation
Istanbul, Turkiye
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Aug 27, 2026Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1LLM-as-a-Judge LLM Agent Evaluation
ServiceNow AI Hyderabad, India
Same Model, Different Harness: Different Coding-Agent Results
Aug 26, 2026Sydney LewisAutomated Agent Evaluation LLM Agent Harnesses
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Aug 18, 2026Liya Zhu, Xin Ma, Tao Liu +35AI Agent Benchmarks LLM Agent Evaluation
ByteDance Seed · Nanjing University · M-A-P
$A^2E$ : An End-to-End Agent Auditing Engine
Aug 7, 2026Haoning Wang, Mingxun Zhang, Chenyue Yu +4Tool-Use Evaluation AI Agent Auditing
Shanghai Artificial Intelligence Laboratory
VAmoS Bench: Voice Agent Simulation Bench
Jul 29, 2026Joshua Meyer, Sahar Shayegan, Ritiz Tambi +5AI Agent Benchmarks Customer Support Automation
Veris AI
The Autonomous Agency Scale: A Behavioral Framework for Measuring Self-Directed Behavior in AI Systems
Jul 20, 2026Samuel PresgravesAI Agent Evaluation AI Agent Benchmarks
Independent researcher.
Copy-on-Write Scoring: Application-Specific Agent Evaluations
Jul 15, 2026Joanna Roy, Sven HoelzelTool-Use Evaluation LLM Agent Evaluation
trail-ml, Munich, Germany
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5LLM Agent Evaluation Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntAI Agent Reliability Agent Evaluation
Independent Researcher
Counsel: A Meta-Evaluation Dataset for Agentic Tasks
Jun 19, 2026Sashank Pisupati, Henry Broomfield, Eujeong Choi +5LLM-as-a-Judge LLM Agent Evaluation
1Atla AI · 2Cohere AI · 3Mistral AI +1
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Jun 16, 2026Zeyao Du, Tong Li, Haibo ZhangWeb Agents Long-Horizon Agent Evaluation
Shopee
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
Jun 16, 2026Congjie Zheng, Chuanyi Xue, Bin Liang +2LLM Agent Evaluation AI Agent Benchmarks
Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
Jun 15, 2026Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh +1Personal AI Agents Tool-Augmented Language Model Agents
Carnegie Mellon University
Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents
Jun 6, 2026Hyogon Ryu, Jeonghwan Kim, Yewon Lim +3LLM-as-a-Judge LLM Agent Evaluation
KRAFTON
RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Computer-Use Agent Benchmarks AI Coding Agents
1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*
PithTrain: A Compact and Agent-Native MoE Training System
May 29, 2026Ruihang Lai, Hao Kang, Haozhan Tang +6Mixture-of-Experts Language Models Efficient Language Model Training
1Carnegie Mellon University · 3NVIDIA · 2Xlue
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerLLM Agent Evaluation AI Agent Benchmarks
IIBM Research
An Empirical Study of Automating Agent Evaluation
May 12, 2026Kang Zhou, Sangmin Woo, Haibo Ding +15Agent Evaluation AI Agent Benchmarks
AWS AI Labs
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses Tool-Augmented Language Model Agents
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8

Oct 16, 2026J/K move · Enter open · S save

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

Haoming Li, Defu Cao, Xu Hu, Yingtao Luo+22

The University of Texas at Dallas · University of Southern California · Carnegie Mellon University+14

Computer-Use Agent Evaluation Computer-Use Agent Benchmarks Computer-Use Agents Agent Evaluation Automated Agent Evaluation Human Preference Evaluation

Open paper AskPDF arXiv

Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html