AI Agent Evaluation

Follow this topic Ask about this topic

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

525 papers77 papers in the last four weeks, up 4% on the four weeks before. 0.4% of all new papers.

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
May 13, 2026Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
May 12, 2026Miaosen Zhang, Xiaohan Zhao, Zhihong Tan +14Computer-Use Agent Benchmarks Computer-Use Agents
1Southeast University · 2Mohamed bin Zayed University of Artificial Intelligence · 3Wuhan University +2
No More, No Less: Task Alignment in Terminal Agents
May 12, 2026Sina Mavali, David Pape, Jonathan Evertz +5Computer-Use Agent Benchmarks AI Alignment
CISPA Helmholtz Center for Information Security · ELLIS Institute Tübingen and MPI for Intelligent Systems · Tübingen AI Center
Rollout Cards: A Reproducibility Standard for Agent Research
May 12, 2026Charlie Masters, Ziyuan Liu, Stefano V. AlbrechtAI Agent Evaluation Agent Evaluation
Deepflow · Nanyang Technological University
Toward Modeling Player-Specific Chess Behaviors
May 12, 2026Loris Sogliuzzo, Aloïs Rautureau, Eric PietteGame-Playing Agents AI Agent Evaluation
ICTEAM, UCLouvain · Louvain-la-Neuve, Belgium
ABRA: Agent Benchmark for Radiology Applications
May 11, 2026Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran +1Radiology Report Generation Healthcare
School of Computing Dublin City University Dublin, Ireland · AXXX · School of Medicine University College Dublin Dublin, Ireland
Generalizing the Turing Test to Interactive Agents
May 11, 2026Daniel Mitropolsky, Riccardo Neumarker, Emanuele Rimoldi +2LLM Evaluation AI Agent Evaluation
MIT · ETH Zurich · EPFL
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
May 11, 2026Pedro Conde, Henrique Branquinho, Valerio Mazzone +3Software Vulnerability Detection AI Agent Evaluation
Ethiack Coimbra, Portugal · Ethiack Porto, Portugal · University of Notre Dame Notre Dame, Indiana, USA
Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability
May 11, 2026Harsh Raj, Niranjan Orkat, Suvrorup Mukherjee +3AI Agent Reliability Agent Reliability
Northeastern University · University of P´ecs · University of Michigan +2
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation
May 11, 2026Shanshan Gao, Liyi ZhouComputer-Use Agent Benchmarks Benchmark Design
The University of Sydney
Agentic Performance at the Edge: Insights from Benchmarking
May 11, 2026Shiqiang Wang, Herbert WoisetschlägerLLM Agent Evaluation AI Agent Evaluation
University of Exeter · Exeter, United Kingdom · Technical University of Munich +1
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
May 11, 2026Haonan Dong, Qiguan Feng, Kehan Jiang +3AI Alignment AI Agent Evaluation
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Key Laboratory of Machine Perception (Ministry of Education), Peking University · School of Software and Microelectronics, Peking University +1
SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems
May 11, 2026Zonglin Yang, Xingtong Liu, Xinyan XuAI Agents for Scientific Discovery AI Agent Evaluation
Readraft Lab · Peking University · Tongji University +1
Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence
May 10, 2026Alex Bogdan, Adrian de Valois-FranklinAI Agent Evaluation AI Agent Monitoring
Evolutionairy AI · Toronto, Canada
MDGYM: Benchmarking AI Agents on Molecular Simulations
May 9, 2026Vinay Kumar, Satyendra Rajput, Mausam +1AI Agents for Scientific Discovery AI Agent Evaluation
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Department of Civil and Environmental Engineering, Indian Institute of Technology Delhi +1
Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?
May 9, 2026Filippo Ziliotto, Ciro Beneduce, Bruno Lepri +3AI Agent Evaluation Vision-Language Grounding
University of Padova · FondazioneLucianoBrunoSerafiniKessler · Fondazione Bruno Kessler
Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents
May 9, 2026Ying Chen, Lihuang Fang, Rui Jiang +4AI Agent Evaluation Agent Evaluation
XPENG ROBOTICS
Log analysis is necessary for credible evaluation of AI agents
May 8, 2026Peter Kirgis, Sayash Kapoor, Stephan Rabanser +8Agent Failure Analysis AI Agent Evaluation
Princeton University · Independent · UK AISI +4
Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
May 8, 2026Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula +4Competitive Analysis Multi-Agent Orchestration
1IBM · 2Artificial Intelligence Institute at University of South Carolina · 3Steel Authority of India Limited +2
Behavioral Determinants of Deployed AI Agents in Social Networks: A Multi-Factor Study of Personality, Model, and Guardrail Specification
May 8, 2026Sarah Wilson, Diem Linh Dang, Usman Ali Moazzam +2AI Agent Evaluation LLM Agents
Columbia University
Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare
May 8, 2026Prasanna Desikan, Harshit Rajgarhia, Shivali Dalmia +1Healthcare Benchmark Design
Centific AI Research Redmond, WA, USA
When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory
May 8, 2026Jiaqi Shao, Yiyi Lu, Yunzhen Zhang +1Agent Memory AI Agent Evaluation
Hong Kong University of Science and Technology, Hong Kong · Duke Kunshan University, China
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
May 8, 2026Yi Liu, TingFeng Hui, Wei Zhang +4AI Agent Evaluation AI Agent Benchmarks
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology · Chongqing University
Computer Use at the Edge of the Statistical Precipice
May 7, 2026Pierluca D'Oro, Sneha Silwal, William Wong +6Computer-Use Agent Benchmarks Computer-Use Agents
Meta Superintelligence Labs
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
May 7, 2026Roger Creus Castanyer, Pablo Samuel Castro, Glen BersethAI Agent Evaluation AI Agent Benchmarks
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1
When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoning
May 7, 2026Vasilis Niarchos, Constantinos Papageorgakis, Alexander G. Stapleton +1AI for Science AI Agent Evaluation
Department of Physics, CCTP and ITCP, University of Crete, 71303, Greece · Centre for Theoretical Physics, Department of Physics and Astronomy, Queen Mary University of London, London E1 4NS, United Kingdom · Theoretical Physics Department, CERN, Geneva, Switzerland +1
Process Matters more than Output for Distinguishing Humans from Machines
May 7, 2026Milena Rmus, Mathew D. Hardy, Thomas L. Griffiths +1AI Agent Evaluation Imitation Learning
Roundtable Technologies Inc. · Princeton University
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
May 7, 2026Jonas Wiedermann-Möller, Leonard Dung, Maksym AndriushchenkoAI Agent Evaluation AI Agent Safety
Universität Bielefeld · Ruhr-Universität Bochum · ELLIS Institute Tübingen +2
BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents
May 7, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +8AI Agents for Scientific Discovery Deep Research Agents
University of Oxford · University College London · Technical University of Munich +1
Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback
May 7, 2026Mohammad Al Ridhawi, Mahtab Haj Ali, Hussein Al OsmanTime Series Forecasting AI Agent Evaluation
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada

May 13, 2026J/K move · Enter open · S save

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose+9

ServiceNow AI Research

Voice Agent Evaluation Spoken Dialogue Systems AI Agent Evaluation Agent Evaluation Speech Processing AI Agent Benchmarks

Open paper AskPDF arXiv

Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversation simulation and comprehensive voice-specific evaluation. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates dynamic bot-to-bot audio conversations with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy) and EVA-X (Experience). EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and multi-trial measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean

\Delta

up to 0.314). We release EVA-Bench under an open-source license.