AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,505 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
May 3, 2026Xiao JiaAI Agent Reliability LLM Agent Evaluation
School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong, Shenzhen
The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't
May 3, 2026Kwan Soo ShinAI Agent Benchmarks LLM Honesty
PolymathMinds AI Lab
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation
May 2, 2026Dong Xu, Jialun Cao, Guozhao Mo +9LLM Agent Evaluation Formal Verification
Institute of Software, Chinese Academy of Sciences · The Hong Kong University of Science and Technology · Guangzhou Institute of Technology, Xidian University
ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue
May 2, 2026Daoxuan Zhang, Ping Chen, Jianyi Zhou +1Aerial Robotics Robotics Simulation
Harbin Institute of Technology, Shenzhen
Can Coding Agents Reproduce Findings in Computational Materials Science?
May 1, 2026Ziyang Huang, Yi Cao, Ali K. Shargh +15Coding Agents AI Agent Benchmarks
Johns Hopkins University
AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
May 1, 2026Ranit Karmakar, Jayita ChatterjeeComputer-Use Agent Benchmarks LLM Agent Evaluation
Harvard University, Boston, MA 02115 · Boston, MA 02115
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent Evaluation AI Agent Benchmarks
1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery
Apr 30, 2026Hanane Nour Moussa, Yifei Li, Zhuoyang Li +7AI Agents for Scientific Discovery AI Agent Benchmarks
1The Ohio State University · 2Cisco Research
KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
Apr 30, 2026Thomas Grady, Kip Parker, Iliyan Zarov +3Sequential Decision Making Long-Horizon Agent Evaluation
General Reasoning, Inc.
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Apr 30, 2026Jinchao Li, Yunxin Li, Chenrui Zhao +3Computer-Use Agent Benchmarks Computer-Use Agents
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
RoadMapper: A Multi-Agent System for Roadmap Generation of Solving Complex Research Problems
Apr 30, 2026Jiacheng Liu, Zichen Tang, Zhongjun Yang +8Multi-Agent LLM Systems LLM Planning
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology (Guangzhou) · IDEA Research +1
ClawGym: A Scalable Framework for Building Effective Claw Agents
Apr 29, 2026Fei Bai, Huatong Song, Shuang Sun +11Synthetic Data Generation AI Agent Benchmarks
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
Apr 29, 2026Yerong Wu, Tianxing Wu, Minghao Zhu +2Conversational Memory AI Agent Benchmarks
School of Computer Science and Engineering, Southeast University, China · 2Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China · 3Independent Researcher +1
LATTICE: Evaluating Decision Support Utility of Crypto Agents
Apr 29, 2026Aaron Chan, Tengfei Li, Tianyi Xiao +3AI-Assisted Decision Making AI Agent Benchmarks
Sahara AI · University of Southern California
DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios
Apr 28, 2026Jinxiang Meng, Shaoping Huang, Fangyu Lei +17Data Visualization AI Agent Benchmarks
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*
ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable
Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-Judge AI Agent Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA
Cooperate to Compete: Strategic Coordination in Multi-Agent Conquest
Apr 28, 2026Abigail O'Neill, Alan Zhu, Mihran Miroyan +2Multi-Agent Coordination Multi-Agent Negotiation
University of California, Berkeley
Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
Apr 27, 2026Joshua Sherwood, Ben Aybar, Benjamin KaplanAI Coding Agents AI Agent Evaluation
Department of Computer Science University of Chicago Chicago, IL 60637 · Independent Researcher
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark Auditing AI Agent Benchmarks
Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1
GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
Apr 27, 2026Yunsu Kim, Kaden Uhlig, Joern WuebkerAI Agent Evaluation AI Agent Benchmarks
LILT
Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft
Apr 27, 2026Zhou Ziheng, Huacong Tang, Jinyuan Zhang +9AI Agents for Scientific Discovery Circuit Discovery
University of California, Los Angeles · Peking University · Amazon
NeuroClaw Technical Report
Apr 27, 2026Cheng Wang, Zhibin He, Zhihao Peng +7Multi-Agent LLM Systems AI Agent Benchmarks
Electronic Engineering Department, The Chinese University of Hong Kong, China · School of Electronic Information, Northwest University, China · Department of Computer Science, Emory University, Atlanta, GA, USA +3
Skill Retrieval Augmentation for Agentic AI
Apr 27, 2026Weihang Su, Jianming Long, Qingyao Ai +6AI Agent Benchmarks LLM Agent Skill Retrieval
Department of Computer Science and Technology, Tsinghua University · ByteDance Inc.
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents
Apr 27, 2026Zheng Wu, Yi Hua, Zhaoyuan Huang +8Computer-Use Agents AI Agent Evaluation
School of Computer Science, Shanghai Jiao Tong University
AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment
Apr 27, 2026Yuxuan Gao, Megan Wang, Yi Ling YuAI Agent Evaluation AI Agent Benchmarks
University of Pennsylvania · 2Columbia University · 3OpenMesh AI

May 4, 2026J/K move · Enter open · S save

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler, Kavita Renduchintala+9

Stanford University

AI Agent Benchmarks Electronic Health Records Long-Horizon Agent Evaluation Healthcare LLM Agents Clinical Language Model Evaluation+1

Open paper AskPDF arXiv

We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.